I
I
Igor Statkevich2019-11-13 14:20:36
Python
Igor Statkevich, 2019-11-13 14:20:36

How to organize multithreading to run subprocess?

There is a need to run subprocess in threads with a controlled number of threads, with the ability to add tasks on the fly.
While it is implemented like this, I start the number of processes from the multiprocessing module equal to the number of processes, I throw the tasks that need to be performed into the queue, as the tasks run out, the threads died, I start new tasks, I look at the previous threads is_alive(), if they are dead, I kill terminate() and start the threads again, if new tasks have been added, the previous ones have not yet completed, they are simply queued and executed by old threads, since they have not yet died.
map is not satisfied with the fact that there is no way to add new tasks, only by creating a new pool.
Is there a more rational solution? To run several subprocesses with a controlled number of threads and the ability to replenish the tasks that need to be run. Data from subprocess is not required, just run in async

Answer the question

In order to leave comments, you need to log in

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question