I
I
Ivan Yakushenko2019-07-10 20:03:14
Python
Ivan Yakushenko, 2019-07-10 20:03:14

How to limit the total number of executions of a multiprocessing script?

The script is run like this:

with Pool(processes=2) as pool:
    for proxy in PROXY_HOST:
        pool.apply_async(registration, (proxy, groups))
    pool.close()
    pool.join()

In this form, it works as long as there are proxies in the list.
I need to limit the number of executions.
1. Passing the status to the database (or any other file) is not an option, because in any case, the thread will be launched - the value in the database will be checked and only then executed/closed.
2. Removing an unnecessary number from the list of proxies is also not suitable, because there may be blocked proxies. That is, for example, I need to make 5 registrations, I add 5 proxies, but 2 of them are blocked, respectively, I will receive only 3 registrations.

Answer the question

In order to leave comments, you need to log in

2 answer(s)
S
Sergey Gornostaev, 2019-07-10
@kshnkvn

You can use Semaphore to control the number of executions.
And by the way, don't close the pool when you use with.

A
alternativshik, 2019-07-10
@alternativshik

check the result via pool.apply_async(registration, (proxy, groups)).get()
if successful regs >= required amount - break

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question