S
S
Sergey Yavin2019-10-02 00:04:26
Python
Sergey Yavin, 2019-10-02 00:04:26

Website parsing Python + Beautifulsoup. A waiting spider?

Tell the student what methods can make the parser wait? Goal: The script should sit on the site until the final data is shown (approximately 10 minutes). Data is added to the table at different time intervals.

Answer the question

In order to leave comments, you need to log in

2 answer(s)
C
coderisimo, 2019-10-02
@sjaserds

Here you need Selenium WebDriver and its "expectations
"

R
r4khic, 2019-10-02
@r4khic

You can use one like this .
Example:

import schedule
import time

def job():
    print("I'm working...")

schedule.every(10).minutes.do(job)
schedule.every().hour.do(job)
schedule.every().day.at("10:30").do(job)
schedule.every().monday.do(job)
schedule.every().wednesday.at("13:15").do(job)
schedule.every().minute.at(":17").do(job)

while True:
    schedule.run_pending()
    time.sleep(1)

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question