S
S
Sergey Ilyin2021-11-28 02:13:30
Parsing
Sergey Ilyin, 2021-11-28 02:13:30

How to parse a page (selenium)?

I opened a question, but the previous one, apparently, drowned in the tape.
Based on previous comments, I was able to throw in code like this:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options


from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.support.ui import WebDriverWait
from time import sleep

# попытка имитировать живого пользователя через хэдэр
chrome_options = Options()
chrome_options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36")

wb_link = webdriver.Chrome(ChromeDriverManager().install(), options=chrome_options)
        
wb_link.get('https://catalog.usmint.gov/coins/coin-programs/america-the-beautiful-quarters-program/')
sleep(2)

# поиск по css селекторам
find_div = wb_link.find_element_by_css_selector("div.product-mint-location").click()

for i in range(3):
  find_div.send_keys(Keys.PAGE_DOWN);
  sleep(5)

# не понимаю - нужен ли вэйт при поиске по селекторам
# WebDriverWait(wb_link, 10).until(lambda driver: driver.execute_script('return document.readyState') == 'complete')


Problem: if you look at the page "live", you can see that it is additionally loaded when scrolling (and then the button also appears).

Question - how to simulate a scroll so that additional elements appear?

Answer the question

In order to leave comments, you need to log in

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question