How to parse multiple pages?

Question

Python

Hello.
Please tell me how I can get certain data from the html page (knowing the element selector), but having 100+ pages. (ala google output).

Answer 1

IMHO, urllib/requests/bs4 is the last century.
Take grab, it has excellent Russian documentation and a user-friendly interface.

Answer 2

requests + BeautifulSoup

Answer 3

urllib2 + BeautifulSoup
The algorithm is as follows (Google issuance), such a pseudo-code:
look at the issue page ,
take all 10 site urls, go
through them all,
opening each one and taking the necessary information using BeautifulSoup,
look at the address of the next Google page,
substitute it at the beginning of the program
And so we continue or until the end all found pages, or specify the desired viewing depth.

Answer 4

In order to iterate over this matryoshka, recursion is needed.