sending multiple cron requests

Question

ruby

there is a service with a list of many addresses of different APIs, it is necessary to send a request there every time interval and write the result to the database. what methods, technologies you used for such tasks or presumably can be used. Now the usual php script running on the crown is working.
when there are many addresses and some do not respond, the script does not have time to work out in the prescribed period of time. therefore it is necessary to parallelize this business.

Answer 1

In light of the fact that the project is being rewritten in Ruby, I would advise you to pay attention to Resque . There is also an alternative in the form of Sidekiq , which can provide a better level of parallelism due to a real fork, so I would suggest starting with it anyway.
In addition to the solutions proposed above, which are more specific to Ruby, you can use another approach: put, for example, RabbitMQ is a message broker written in Erlang. The scheme of working with RabbitMQ, simplified, is as follows:
there are queues controlled by RabbitMQ; different processes can write messages to these queues (for example, by cron, they launched a script that determined the scope of work, and sent messages about the work to the queues); there are processes that “listen” to a certain queue, and when a new message arrives in the queue, RabbitMQ selects one listening process and passes this message to it (that is, RabbitMQ distributes messages in the queue among processes).

Answer 2

gearman - simple and fast.

Answer 3

If you need a bike, then here it is:
Divide your script into 2 parts, in the first logic into the second api list and the “launcher” of the logic with this list.
You can split the second part of the script into any desired amount.
Offtopic. Why the list of hubs ruby ​​=)?

Answer 4

First, we twist the timeouts correctly. Does not answer and does not have time to work out - these are generally different things.
If you really don’t want to rewrite anything or don’t have time, you can look at xargs -n1 -P