B
B
beginer1232016-09-18 18:37:36
Python
beginer123, 2016-09-18 18:37:36

Python multiprocessor parser hangs?

Task: to parse the site, make each request from different ip, to speed up all this I use multiprocessing
By steps I
create a pool I create a mass of url-s I apply it to the parsing function However, it parses somewhere around a third of the urls, then it just thinks and that’s it . That is, my script does not end with code 1 or 0 What could be the reason?
pool = Pool(processes=5)
urls = [....]
results = pool.map(parse_func, urls )

Answer the question

In order to leave comments, you need to log in

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question