Run a task periodically?

Question

Ruby on Rails

Hello. Tell me please. How can I solve this problem correctly. You need an http request to pull data from the site and save it to the database. This is implemented, but how would the question be that it needs to be done every minute?
How to use the whenewer gem , then it turns out with frequent requests when one is in the process of being executed, and the other has started, there may be problems with locks?
The threads advised, like through the gem ' cuncurrent-ruby ' Concurrent::TimerTask , but here it’s like if you embed it in a rail application, then it will start its own thread for each Worker. And if you write a separate one, it is not clear how best to do all this?
Tell me, can someone similar implemented?

Answer 1

There is some confusion
here. Processes are launched through cron, not threads, so there should be no problems with locks, because each launch is its own process with its own address space. There may be data integrity issues in the database.
Now about the implementation:
This is best done through a queue service like sidekiq.
How it works?
Through the sidekiq API, you can queue a task (you can also put it once a minute, this can be specified in config), launch parameters are stored in redis (which will need to be put on production). The parsing task queue needs to be configured for 1 thread of execution.
Then sidekiq, hanging out as a separate process besides rails, simply takes the topmost task from the queue and executes it.
Thus, if the parsing is more than a minute, the queue length simply increases. This value should be monitored, but if it grows, there will be no catastrophe.
You can probably do it both through ActiveJob and in the main database - Rails, I have not yet looked at the innovations about this.

Answer 2

Cron adds a task to the queue. The task in the queue should take as little time as possible.