A
A
Alena_Y2020-10-24 18:11:50
Python
Alena_Y, 2020-10-24 18:11:50

How to scrape dynamic content with scrapy?

Hello, I needed to scrape google images using scrapy, but I can't figure out how to scroll the page to download the next photos using scrapy.
There is an implementation on scrapy + selenium, but such a bundle is rather slow
. Is it possible to scroll the site using scrapy?

Answer the question

In order to leave comments, you need to log in

1 answer(s)
S
soremix, 2020-10-24
@SoreMix

Scroll - no, it's not a browser.
Open the developer tools on F12, open the Network tab, scroll the page, see what requests have been sent for new images, most likely they will be in the XHR tab. And repeat them through python

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question