R
R
ronny2021-11-23 10:48:05
Python
ronny, 2021-11-23 10:48:05

Not a complete answer in python requests?

I want to parse data using this link https://www.mywot.com/scorecard/vk.com

The CURL option works fine - the entire page is returned as in a browser

curl -X GET -k -i -v 'https://www.mywot.com/scorecard/vk.com'


The completeness of the page can be checked by the content in it, for example, a line. __WOT__

But when a request is made through python, the page is returned incomplete, although with a closed /body.

requests.get('https://www.mywot.com/scorecard/vk.com')


Yes, I know about headers and user agent. But note that the curl has standard headers. And everything is ok. And with python, even with the substitution of the user agent, it is still a strange incomplete page.

Can someone come across this?

Answer the question

In order to leave comments, you need to log in

1 answer(s)
S
soremix, 2021-11-23
@ronny

>>> '__WOT__' in requests.get('https://www.mywot.com/scorecard/vk.com').text
True

?
requests returned HTML length 454779, length with CURL is 454749

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question