A
A
Alex Wells2018-08-05 04:07:36
linux
Alex Wells, 2018-08-05 04:07:36

Run N different queues, X each per Z threads?

Hello. There is a need to almost dynamically create a dozen queues, each of which once a second receives a task for ~ 50ms of work. A specific application is to manipulate different tables in mysql one by one. It would seem that you can create all the queues right in the same process - but then the dulling of one queue will lead to problems with the other, and this is not an option, so the task is to parallelize all this into different threads, for example, child_process.
Question: what happens if I run 20 queues in 20 sub-processes on an 8-threaded CPU, the task of each of which is to perform asynchronous operations (that is, without wasting CPU time)? How is the load distributed among the cores? How good is this approach? Will I have to write some kind of distribution system for sub-processes to process several queues in one process, so that there are 8 of them in total?

Answer the question

In order to leave comments, you need to log in

3 answer(s)
S
Stanislav Bodrov, 2018-08-05
@jenki

perform asynchronous operations (that is, without wasting processor time)
Where do you think they are processed? In the south bridge?
How is the load distributed among the cores?
How you set up the server: NUMA, OS scheduler, third-party processes on the server, target application settings.
In MySQL, one request is one process. What will be inside the process is at the mercy of the MySQL scheduler.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question