Answer the question
In order to leave comments, you need to log in
Python requests, when displaying the entire site, displays the body and after it: Request unsuccessful. How to fix?
After outputting soup, the output is:
Request unsuccessful. Incapsula incident ID: 534000470010449392-29143315938413509
And at the end: Request unsuccessful. Incapsula incident ID: 534000470010449392-29143315938413509 Therefore, I constantly cannot compile the code, either it downloads the site or not. I already turned on the proxy, I thought that they were banning bots, but nothing stopped
import requests
from bs4 import BeautifulSoup
http_proxy = "http://128.199.239.230:8080"
proxyDict = {
"http" : http_proxy
}
url = "https://eldorado.ua/uk/vytjazhka-gunter-hauer-agna-600-gl/p71226038/"
Headers = {"user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36", "accept": "*/*"}
def get_html(url , params=None):
r = requests.get(url, headers = Headers, params = params, proxies = proxyDict)
return r
def get_content(html):
soup = BeautifulSoup(html, "html.parser")
print(soup)
items_e = soup.find_all("div", class_="product-head-text")
def parse ():
html = get_html(url)
if html.status_code == 200:
get_content(html.text)
else:
print("Error")
if __name__ == '__main__':
parse()
Answer the question
In order to leave comments, you need to log in
from selenium import webdriver
driver = webdriver.Firefox()
# driver = webdriver.Chrome()
url = 'https://eldorado.ua/uk/vytjazhka-gunter-hauer-agna-600-gl/p71226038/'
driver.get(url)
html = driver.page_source
driver.quit()
print(html)
Encapsule is a popular cdn, protected from bots, ddos and more. Actually, it protects the site from this.
Unless to look for detours / write your own
I understand that writing your own bypass is difficult, so there is no other solution?
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question