A
A
activebomb2019-08-05 13:36:16
Python
activebomb, 2019-08-05 13:36:16

How to parse from multiple pages?

Look, now I'm parsing the result from only one page, let's say - https://www.google.ru/search?q=%D0%BA%D0%B0%D1%80%... https://www. google.com/search? from this page.
I need to parse the result from all the pages that exist, that is, start=10, so that after the page has been parsed, 10 more will be added to start=10.
code example:

def bf_parse(base_url, headers):
    session = requests.session()
    request = session.get(base_url, headers=headers)
    if request.status_code == 200:
        soup = bs(request.content, 'html.parser')
        links = (d.find('a') for d in soup.find_all('div', attrs=('goodsDescription')))
        users = {a.get('href') for a in links if a is not None}
        f = open("user_baza.txt", "a")
        f.write('\n'.join(users))
        f.close()
        print('\n'.join(users))
    else:
        print("ерор")


bf_parse(base_url, headers)

Answer the question

In order to leave comments, you need to log in

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question