Python Web: Multiprocessing vs. Threads. What is better to use for parsing?

Question

Python

Apparently I do not understand the difference too much, but I do not understand - which is better to use for web parsing? Is there any Russian reading on this topic?

Answer 1

You can use Scrapy . Then you don't have to think about parallel processes, locks and IO in general. You just write the logic for parsing the page. The project itself is on Twisted.

Answer 2

One thing I want to say about lxml - do not use it to parse with the URL passing as the source. It is better to download the page with the same request and drive it into lxml via document_fromstring - save yourself a lot of nerve cells.