Answer the question
In order to leave comments, you need to log in
Python multiprocessor parser hangs?
Task: to parse the site, make each request from different ip, to speed up all this I use multiprocessing
By steps I
create a pool I create a
mass of url-s I
apply it to the parsing function
However, it parses somewhere around a third of the urls, then it just thinks and that’s it
. That is, my script does not end with code 1 or 0
What could be the reason? pool = Pool(processes=5)
urls = [....]
results = pool.map(parse_func, urls )
Answer the question
In order to leave comments, you need to log in
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question