D
D
dima_program2020-06-17 10:16:00
Python
dima_program, 2020-06-17 10:16:00

How to scroll down a page in python?

Good afternoon everyone. There is a parser that parses this page:
https://www.flashscore.ru/
It clicks on matches and displays their link (the site is designed in such a way that it opens the second Google Chrome and this match is already in it, run the script and you will understand what I mean say). But after the 17th match (LASK/Shturm) displays an error. You need to scroll down the page to click on matches further. Tell me how to scroll the page down a few matches or even to the very bottom? Here using js to do it? Thanks in advance.
The code:

import time
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from bs4 import BeautifulSoup

count = 0
driver = webdriver.Chrome(executable_path="C:\\Users\\iljal\\PycharmProjects\\google_sheets\\chromedriver") # здесь указать путь к chromedriver он обычно в той же папке где и Ваш проект
driver.get('https://www.flashscore.ru/')
time.sleep(3) # C:\Users\iljal\PycharmProjects\google_sheets\chromedriver

arr = driver.find_elements_by_css_selector(
    ".event__match.event__match--scheduled.event__match--oneLine"
)


# optional (if you are not satisfied with the download speed)
driver.set_page_load_timeout(0.5)


for channel in arr:
    try:
        channel.click()
        count += 1
        if count == 17:
            driver.execute_script(
                "var evt = document.createEvent('MouseEvents');evt.initEvent('wheel', true, true);evt.deltaY = -100000;document.querySelector('.yamb-conversation__content').dispatchEvent(evt);") # такая прокрутка не работает, выводит ошибку:
    except TimeoutException:
        print("data not received. need more time in driver.set_page_load_timeout")
        continue

    driver.switch_to.window(driver.window_handles[arr.index(channel)+1])


    link = driver.current_url
    print(link)

    driver.switch_to.window(driver.window_handles[0])

After 17 matches, it throws an error when trying to scroll down:
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.JavascriptException: Message: javascript error: Cannot read property 'dispatchEvent' of null
  (Session info: chrome=83.0.4103.97)

Answer the question

In order to leave comments, you need to log in

2 answer(s)
S
soremix, 2020-06-17
@SoreMix

driver.execute_script("window.scrollTo(0, 1080)")
or

driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

D
Dmitry, 2020-06-17
@LazyTalent

In this case, IMHO, it's better to scroll the page not to the very bottom, but to the element you want to click on:

for channel in arr:
    try:
        driver.execute_script("arguments[0].scrollIntoView();", channel)
        channel.click()

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question