Y
Y
Yurii Abraham2021-07-13 20:26:00
Python
Yurii Abraham, 2021-07-13 20:26:00

Python requests, when displaying the entire site, displays the body and after it: Request unsuccessful. How to fix?

After outputting soup, the output is:

Request unsuccessful. Incapsula incident ID: 534000470010449392-29143315938413509
And at the end: Request unsuccessful. Incapsula incident ID: 534000470010449392-29143315938413509 Therefore, I constantly cannot compile the code, either it downloads the site or not. I already turned on the proxy, I thought that they were banning bots, but nothing stopped

import requests

from bs4 import BeautifulSoup

http_proxy  = "http://128.199.239.230:8080"

proxyDict = {
              "http"  : http_proxy
            }

url = "https://eldorado.ua/uk/vytjazhka-gunter-hauer-agna-600-gl/p71226038/"
Headers = {"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36", "accept": "*/*"}

def get_html(url , params=None):
    r = requests.get(url, headers = Headers, params = params, proxies = proxyDict)
    return r

def get_content(html):
    soup = BeautifulSoup(html, "html.parser")
    print(soup)
    items_e = soup.find_all("div", class_="product-head-text")

def parse ():
    html = get_html(url)
    if html.status_code == 200:
        get_content(html.text)
    else:
        print("Error")

if __name__ == '__main__':
    parse()

Answer the question

In order to leave comments, you need to log in

3 answer(s)
A
ak545, 2021-07-14
@Yura_nu_davai_rabotai

from selenium import webdriver

driver = webdriver.Firefox()
# driver = webdriver.Chrome()
url = 'https://eldorado.ua/uk/vytjazhka-gunter-hauer-agna-600-gl/p71226038/'
driver.get(url)
html = driver.page_source
driver.quit()

print(html)

60ee307667c72496351911.png

S
soremix, 2021-07-13
@SoreMix

Encapsule is a popular cdn, protected from bots, ddos ​​and more. Actually, it protects the site from this.
Unless to look for detours / write your own

Y
Yurii Abraham, 2021-07-13
@Yura_nu_davai_rabotai

I understand that writing your own bypass is difficult, so there is no other solution?

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question