How to skillfully work with Sidekiq and queues?

Question

Ruby on Rails

I struggle for a very long time and I can not understand the essence of the work. What I want to do: work with instagram (subscriptions and likes), select hashtags and our application starts working, we do several tasks for different accounts; in principle it works, but...
SIdekiq launches all workers and jobs at startup, which is not required at the very beginning. How to correctly call and add queues. I struggle for a long time, I can’t understand (I read the manual and something doesn’t work).

Answer 1

Briefly about the essence - in general, any site should return the page to the client < 3 seconds. That is, rails should quickly process the request and return a response. Sending mail or "working with instagram" are quite complex tasks to handle, which will slow down the speed of the HTTP response. If the "hard" part of the work is given to sidekiq, then this problem will not arise.
Sidekiq uses Redis to store queues.
In the /app/workers/ folder you place the workers you need. For example:

# app/workers/hard_worker.rb
class HardWorker
  include Sidekiq::Worker

  def perform(name, count)
    puts 'Doing hard work'
  end
end

(the perform method as an entry point is required, the file and class names must end with 'worker', and of course include Sidekiq::Worker is also needed).
When sidekiq is running, you can call from anywhere in the project code:
Which will create a new "job" for sidekiq and add it to the queue. Actually, as the jobs are processed, sidekiq will complete the task assigned to it - it will get it out of the queue and the worker will execute it. More workers - faster processing, the number of workers can be set as an argument when running sidekiq.
PS Most likely you made a mistake in the worker or make a call incorrectly. Please add your code to the question.
+ check railscasts on this topic railscasts.com/episodes/366-sidekiq

Answer 2

Queues are, as bmalets already said , sort of like task lists. They are needed mainly for load control, parallelism. By default, there is only the default queue.
A classic example of splitting queues: - mailer
- a queue for sending mail messages to users
- default - for everything else
make the mailer queue a priority.
Load sharing can also be controlled using "weights" (option -q mailer,9 default,1 ). In this case, out of 10 tasks, 9 will be taken from the mailer queue and only one from the default queue.
Workers can mean two concepts:
- a worker as an OS process that takes tasks from the queues for execution
- a worker as a ruby ​​class that performs some kind of business task
Basically, of course, the second option is meant by workers.
For your questions:
- how are workers executed?
The handler process takes a task from redis and executes it. It is important to understand here that if the process crashes, you can lose the task (such cases can be very critical for business).
- how much memory is needed for this?
The amount of data in redis can be viewed in the admin interface.
- why redis
Who knows...
- how does it work with redis
???
- why in perform_async it is impossible to transfer the objects themselves? but only id, strings, json...?
It is possible, but not necessary. Everything that gets into perform_async should get into redis. And this is the overhead for storage, serialization / deserialization.
- How to correctly call and add queues.
What do you mean by "call queues"? Queues are added automatically
In general, the storage format can be easily viewed in the radish itself.