Why is the entire html code (requests) not displayed?

K

Kirill2020-04-23 16:57:29

Python

Kirill, 2020-04-23 16:57:29

When getting a request via requests, the main part of the resulting html code of the site disappears.

header = { 
  'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36',
  'Content-Type':'application/x-www-form-urlencoded',
  'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
  'Accept-Language':'ru-ru,ru;q=0.8,en-us;q=0.5,en;q=0.3',
  'Accept-Encoding':'gzip, deflate',
  'Connection':'keep-alive',
  'DNT':'1'
}

reg_url = 'https://foxford.ru/user/registration'

session = requests.session()
    data = session.get(reg_url, headers=header).content
    page = lxml.html.fromstring(data)

    print(data)

Received code:

<!DOCTYPE html><html><head><meta charset="utf-8" /><meta content="!" name="fragment" /><meta content="IE=edge" http-equiv="X-UA-Compatible" />
<script></script> Бла-бла-бла скрипты
  <!-- Google Tag Manager -->
  <noscript><iframe src="//www.googletagmanager.com/ns.html?id=GTM-NDCLX6"
                    height="0" width="0" style="display:none;visibility:hidden"></iframe></noscript>
  <script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':
      new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],
      j=d.createElement(s),dl=l!='dataLayer'?'&l='+l:'';j.async=true;j.src=
      '//www.googletagmanager.com/gtm.js?id='+i+dl;f.parentNode.insertBefore(j,f);
  })(window,document,'script','dataLayer','GTM-NDCLX6');</script>
  <!-- End Google Tag Manager -->
<script></script

    <b><i><div id="BaseApp-react-component-40e0b07f-cd07-49bc-b5ed-f15351755f0c"></div></i></b>
    
<script src="https://s.foxford.ngcdn.ru/js-libs/[email protected]/MathJax.js?config=TeX-AMS_SVG" id="MathJax-script" async="async"></script><script src="https://a.foxford.ngcdn.ru/assets/webpack/js/runtime.deaad9a6.js" defer="defer"></script>
<script src="https://a.foxford.ngcdn.ru/assets/webpack/js/vendors.87c41ba3.chunk.js" defer="defer"></script>
<script src="https://a.foxford.ngcdn.ru/assets/webpack/js/common.ui.08fe203f.chunk.js" defer="defer"></script>
<script src="https://a.foxford.ngcdn.ru/assets/webpack/js/lms.06bd8984.chunk.js" defer="defer"></script></body></html>

Namely
<div id="BaseApp-react-component-40e0b07f-cd07-49bc-b5ed-f15351755f0c"></div>
absolute empty. Although it should contain a registration form.

Reply

Answer the question

In order to leave comments, you need to log in

1 answer(s)

D

Dmitry, 2020-04-23
@kustiktm_kirill

Because javacscript is not executed in your case, but is executed in the browser.