E
E
Evgin5552020-06-10 20:59:57
Python
Evgin555, 2020-06-10 20:59:57

How to parse information from the site if there is pagination?

How can this script be improved? The problem is that only one page is parsed. How can I go to the next page and save the phone there too?

from selenium import webdriver
from time import sleep
from PIL import Image
from pytesseract import image_to_string
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
class Bot:
    def __init__(self):
        self.driver = webdriver.Firefox()
        self.navigate()
        wait = WebDriverWait(self.driver, 10)

    def take_screenshot(self):
        self.driver.save_screenshot('avito_screenshot.png')

    def tel_recon(self):
        image = Image.open('tel.png')
        print(image_to_string(image))

        f = open('number.txt', 'a')
        f.write(image_to_string(image)+ '\n')
        f.close()

        def crop(self, location, size):
        image = Image.open('avito_screenshot.png')
        x = location['x']
        y = location['y']
        width = size['width']
        height = size['height']

        image.crop((x, y, x+width, y+height)).save('tel.png')
        self.tel_recon()

        def navigate(self):
        self.driver.get('https://www.avito.ru/volgogradskaya_oblast_volzhskiy/avtomobili/gaz_gazel_2747_2007_1903589701')

        button = self.driver.find_element_by_xpath('//a[@class="button item-phone-button js-item-phone-button button-origin contactBar_greenColor button-origin_full-width button-origin_large-extra item-phone-button_hide-phone item-phone-button_card js-item-phone-button_card contactBar_height"]')
        button.click()

        sleep(15)

        self.take_screenshot()

        image = self.driver.find_element_by_xpath('//div[@class="item-phone-big-number js-item-phone-big-number"]//*')
        location = image.location   # dict {'x': 2343, 'y': 23423}
        size = image.size           # dict {'width': 234, 'height': 234}

        self.crop(location, size)           

def main():
    b = Bot()

if __name__ == '__main__':
    main()

Tried explicit wait, wait, loops.

Answer the question

In order to leave comments, you need to log in

1 answer(s)
S
Sedmi, 2020-06-10
@Sedmi

As an option, collect a list of links to bypass into a file and start parsing through line enumeration.

with open('urls.txt', 'r') as f:
for i in f:
....
self.driver.get(str(i))
...

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question