S
S
SHADRIN2019-02-20 15:09:08
Python
SHADRIN, 2019-02-20 15:09:08

I am learning to parse the site, it does not display anything, what should I do?

import requests
from bs4 import BeautifulSoup as bs

headers = {'accept': '*/*',
           'user-agent': 'Mozilla/5.0(X11;Linux x86_64...)Geco/20100101 Firefox/60.0'}
base_url = 'https://www.scoreboard.com/ru/football/#/'
out = 'output.txt'

def sb_parse(base_url, headers):
    session = requests.session()
    request = session.get(base_url, headers=headers)
    if request.status_code == 200:
        soup = bs(request.content, 'html.parser')

        divs = soup.find_all('table', attrs={'class': 'soccer'})
        
        for div in divs:

            href = div.find('span', attrs={'padr'}).text
            print(href)
    else:
        print('ERROR')

sb_parse(base_url, headers)

It just doesn’t display anything :(
I’m just learning, I asked a similar question, but at least there was an error, it’s just silence.
PS don’t look at the search blocks, I just inserted everything, I thought I was searching in the wrong place
At that time there was JS on the site and it was impossible to parse through bs4 this time I took it without JS (this is not accurate) and the answers are empty, the name of the buttons on the site can be parsed, but there are no matches
If here, too, JS, give excellent documentation on "selenium"

Answer the question

In order to leave comments, you need to log in

3 answer(s)
V
Vlad Grigoriev, 2019-02-20
@Vaindante

Chrome has a useful DevTools thing, it has a Network tab, if you go to it, you will see that a bunch of requests are leaving and in one of them you can see the data for the table:
Request URL: https://d.scoreboard.com/ru/ x/feed/f_1_0_5_ru_1
but this does not mean that you will also just receive an answer, at a cursory glance it looks like you first need to make a session.
and there is also a web socket) and data is running there. you can connect to it
And note that there is Accept-Encoding: gzip, deflate, br

D
Dmitry, 2019-02-20
@LazyTalent

toddhayton.com/2015/03/11/scraping-ajax-pages-with...

A
alternativshik, 2019-02-20
@alternativshik

there the table is loaded by ajax, of course, from BS there is no sense here. See where it comes from and in what form and take it to another address.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question