How to control the work of script-workers? What is better Crontab?

Question

linux

Hello,
I have 6 services written in PHP and Golang that perform one big task.
In order for subtasks to be executed sequentially, we use queues, the first service writes data to the queue, the second service picks them up, and so on, the 6th service completes the cycle and the task is considered completed.
Services are launched using crontab, using a record in the database, we control the status of the service so as not to restart it if it is still running.
This method has minutes, all services are started once a minute.
1st start of services:

• 1st service fulfilled - wrote down in queue.
  
• The 2nd service started idle.
  
• The 3rd service started idle.
  

2nd start of services:

• The 1st service started idle.
  
• The 2nd service worked - wrote down in queue.
  
• The 3rd service started idle.
  

3rd start of services:

• The 1st service started idle.
  
• The 2nd service started idle.
  
• The 3rd service has completed - the task is completed.
  

and the same situation with 4.5 and 6 services.
As a result, up to 7 minutes pass until the task is completed and the information is published on the site.
I wanted to ask if there is a better way to launch and monitor services?
NOHUP with a loop looks interesting, but how do you control it?
How to control that the script is not dead, that it is not run several times?
I would be very grateful if you tell me in which direction to look for a solution to the problem.

Answer 1

• systemd
  
• Supervisor
  

Answer 2

If you need to run them sequentially, why are you running each service in cron separately from the others?
Make a script, enter all the services in turn into it and run the script every 10 minutes?

Answer 3

Alternatively, if the services are "falling", put them in the task scheduler.
the same RabbitMQ. Do 6 turns. For each at least 1 consumer.
The task is thrown in the 1st queue. The first consumer starts processing it. When processing, it places the task in the next queue. In case of a fail, the task will be processed again at the same step - there is a mode when the task returns itself to the queue.
The RabbitMQ service is spinning on the server, consumers can be launched by cron so that they always hang as residents. Or via flock, or daemons via supervisor.

Answer 4

When you have a message queue, you want programs that run once and read and process messages from the queue in a loop. Well, and a supervisor over them, who will pick them up if they fall.
Those. your services should not be launched by cron, but should be constantly running.

Answer 5

"Advanced" crons:
- Rundeck https://www.rundeck.com/open-source
- Airflow https://airflow.apache.org/

Answer 6

There are several ways, I will list as the "correctness" decreases in my opinion:
1) rewrite the services so that they can be constantly running. This will then become a typical architecture, already implemented many times by people before :) Ie. services are running 24/7 and throw information into each other through the queue. You can also read about sagas: https://habr.com/ru/company/oleg-bunin/blog/418235/
2) switch from cron to systemd timers and services. Implement the launch of the first on a timer, and then write dependencies through systemd and it will start everything one after another.
3) use "smart" rundeck type cron, well, this was already advised above.
PS And if this is one, in fact, task, why is it in several applications? Perhaps it's worth collecting them into one thing, go'shnoe for example, and run it when required.