J
J
James_Obry2022-01-27 23:49:25
Python
James_Obry, 2022-01-27 23:49:25

How to parse dynamic text in python?

I am writing a small exchange rate parser in python and the requests and beautifulsoup4 libraries.
Here is the site I want to parse.
Here is the parse code for the tr I need

def parse():
    URL = 'http://sky.bank/uk/currency'
    HEADERS = {
        'user-agent': 'my user agent'
    }
    response = requests.get(URL, headers=HEADERS, verify=False)
    soup = BeautifulSoup(response.content, 'html.parser')
    items = soup.findAll('tr')
    print(items[1])
    print(items[2])
    print(items[3])


If you call this function, you will see that there are 3 tags inside tr 1-currency, 2-buy rate, 3-sell rate. You can also see that each of the buy rates has the numbers 33.6, and each of the sell rates has the numbers 33.9, but on the site there are completely different numbers, as far as I understand, these data are dynamic. So the question is: how to parse these numbers correctly?

Answer the question

In order to leave comments, you need to log in

1 answer(s)
V
Vindicar, 2022-01-28
@Vindicar

They are updated by the script for sure. And the script probably makes a background request to the backend of the site.
From here:
a) Find out what kind of request it is and learn how to execute it yourself using requests. Bonus - it's probably something human-readable, like JSON, not HTML.
b) Use a full-fledged browser that can execute scripts, such as selenium. Cumbersome and slow, but it allows you to successfully mow for the average user.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question