Answer the question
In order to leave comments, you need to log in
How to load data on the site, if when scrolling it rests on "Show more"?
It is necessary to collect the data of all people from the search in the classmate.
The code works until the "Show more" button appears.
I can't get past this place to load the rest of the people. How can this problem be solved?
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
from bs4 import BeautifulSoup
driver = webdriver.Chrome("D:\chromedriver")
driver.get('https://www.ok.ru/dk?st.cmd=anonymMain')
def test_01():
field_email = driver.find_element_by_id('field_email')
field_email.send_keys('логин')
field_password = driver.find_element_by_id('field_password')
field_password.send_keys('пароль')
p = driver.find_element_by_class_name('mt-5x')
p.click()
driver.get('https://ok.ru/search?st.mode=Users&st.grmode=Groups&st.posted=set&st.query=' + 'Петров')
SCROLL_PAUSE_TIME = 3
last_height = driver.execute_script("return document.body.scrollHeight")
while True:
# Scroll down to bottom
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
# Wait to load page
time.sleep(SCROLL_PAUSE_TIME)
# Calculate new scroll height and compare with last scroll height
new_height = driver.execute_script("return document.body.scrollHeight")
if new_height == last_height:
break
last_height = new_height
time.sleep(2)
soup = BeautifulSoup(driver.page_source, "lxml")
divs = soup.findAll("div", {'class': 'gs_result_i_w'})
divs_number = soup.findAll("div", {'class': 'portlet_h_name_t'})
print(divs_number)
for div in divs:
div_title = div.find("div", {'class': 'ellip'})
link_text = div_title.find('').text
print('ФИО : ' + link_text)
test_01()
Answer the question
In order to leave comments, you need to log in
look for the scroll code down
and, well, you found it
, what's the problem then? at each step you parse the body, not at the end
You can simulate a button press click through a call to JS
driver.execute_script('return document.getElementsByClassName("button").click()')
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question