M
M
Maxim Belozerov2022-01-06 14:29:51
Python
Maxim Belozerov, 2022-01-06 14:29:51

How to force selenium to load the part of the page that is loaded after the scrollbar is scrolled?

Let's say I need to parse a page: page .
Selenium only gives out 18 olympiads out of 251. How can I make it load the page completely?
The code:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from bs4 import BeautifulSoup

ser = Service(r'C:\Users\DNS\Desktop\Python\Test\Проект\Driver\chromedriver.exe')
options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--normal')
browser = webdriver.Chrome(service=ser, options=options)

browser.get('https://olimpiada.ru/activities?type=any&class=9&period_date=&period=year&perechen=on')
requiredHtml = browser.page_source

soup = BeautifulSoup(requiredHtml, 'lxml')
open("Htm.html", "w", encoding="utf-16").write(str(soup))

Answer the question

In order to leave comments, you need to log in

1 answer(s)
K
Kadabrov, 2022-01-06
@Kadabrov

you can scroll through js code (available on the Internet)
you can do it through the selenium driver press the 'page dawn' button
how to find out how many screens to scroll, think for yourself

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question