S
S
Sergey Ilyin2021-11-23 03:50:48
Python
Sergey Ilyin, 2021-11-23 03:50:48

How to defeat the parsing of this page in selenium?

there is a page:

https://catalog.usmint.gov/coins/coin-programs/america-the-beautiful-quarters-program

I'm trying to parse it:

from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.ui import WebDriverWait
from time import sleep

wb_link = webdriver.Chrome(ChromeDriverManager().install())
        
wb_link.get('https://catalog.usmint.gov/coins/coin-programs/america-the-beautiful-quarters-program/')
sleep(2)
        
wb_link.execute_script("window.scrollTo(0, 100);")
sleep(1)

wb_link.execute_script("window.scrollTo(100, 500);")
sleep(1)
wb_link.execute_script("window.scrollTo(500, 1000);")
sleep(1)
wb_link.execute_script("window.scrollTo(1000, 1500);")
sleep(1)
wb_link.execute_script("window.scrollTo(1500, 2000);")
        
WebDriverWait(wb_link, 10).until(lambda driver: driver.execute_script('return document.readyState') == 'complete')


everything is simple here:
1. I pass the link to selenium (through a small library, but it doesn’t matter)
2. I scroll through the page (I scroll in several passes - crooked, but an attempt to imitate the user)
3. Using `WebDriverWait` I wait for the page to load.

The trouble is that when I scroll live in the browser - everything is ok. When I start scrolling through selenium, the site does not want to upload pictures, slipping trash there (no, they are great, of course, no doubt, but I want to collect data)

I would be grateful for the tips.

Answer the question

In order to leave comments, you need to log in

1 answer(s)
C
calculator212, 2021-11-23
@calculator212

For the sake of interest, I decided to try it, everything is fine with me, only I used mozilla. So most likely the problem is not in the site.

from selenium import webdriver
from time import sleep

driver = webdriver.Firefox("./")
driver.get('https://catalog.usmint.gov/coins/coin-programs/america-the-beautiful-quarters-program/')

sleep(2)

driver.execute_script("window.scrollTo(0, 100);")
sleep(1)

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question