O
O
OldSpice2019-04-20 21:34:01
Python
OldSpice, 2019-04-20 21:34:01

How to load data on the site, if when scrolling it rests on "Show more"?

It is necessary to collect the data of all people from the search in the classmate.
The code works until the "Show more" button appears.
I can't get past this place to load the rest of the people. How can this problem be solved?

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
from bs4 import BeautifulSoup

driver = webdriver.Chrome("D:\chromedriver")
driver.get('https://www.ok.ru/dk?st.cmd=anonymMain')

def test_01():
  field_email = driver.find_element_by_id('field_email')
  field_email.send_keys('логин')
  field_password = driver.find_element_by_id('field_password')
  field_password.send_keys('пароль')

  p = driver.find_element_by_class_name('mt-5x')
  p.click()

  driver.get('https://ok.ru/search?st.mode=Users&st.grmode=Groups&st.posted=set&st.query=' + 'Петров')

  SCROLL_PAUSE_TIME = 3
  last_height = driver.execute_script("return document.body.scrollHeight")
  while True:
      # Scroll down to bottom
      driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

      # Wait to load page
      time.sleep(SCROLL_PAUSE_TIME)

      # Calculate new scroll height and compare with last scroll height
      new_height = driver.execute_script("return document.body.scrollHeight")
      if new_height == last_height:
          break
      last_height = new_height

   time.sleep(2)
   soup = BeautifulSoup(driver.page_source, "lxml")
   divs = soup.findAll("div", {'class': 'gs_result_i_w'})
  
   divs_number = soup.findAll("div", {'class': 'portlet_h_name_t'})
   print(divs_number)
  
   for div in divs:
       div_title = div.find("div", {'class': 'ellip'})
       link_text = div_title.find('').text
       print('ФИО : ' + link_text)

test_01()

Answer the question

In order to leave comments, you need to log in

2 answer(s)
D
Dimonchik, 2019-04-20
@dimonchik2013

look for the scroll code down
and, well, you found it
, what's the problem then? at each step you parse the body, not at the end

A
Alexander Shilov, 2019-04-26
@tabbols95

You can simulate a button press click through a call to JS

driver.execute_script('return document.getElementsByClassName("button").click()')

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question