Python. I can't parse the price tag in the "DNS" store. Can you help?

Question

Python

import requests
from bs4 import BeautifulSoup

def get_html(url):
    headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'}
    r = requests.get(url, headers = headers)
    return r.text

def get_elements(html, tag, class_name):
    soup = BeautifulSoup(html, 'html.parser')
    pages = soup.find_all(tag, class_= class_name)
    return pages

class DNS():
    html = get_html('https://www.dns-shop.ru/catalog/17a899cd16404e77/processory/order=1&groupBy=none&brand=amd&stock=2')
    price = get_elements(html, 'div', 'product-min-price__current')
    print(price)

DNS()

An empty list is returned! Although the tag class is correct.


If I try to assemble another div that is several layers below, it will output something like this... Where are the rest of the layers that are nested in this class?



Is it some kind of protection? How to bypass it?

Answer 1

Discover postman. It is a handy tool for making queries. And in it send a request to the link that you have. I don’t have the opportunity to check right now, but I think that JS is running on the page, which fills the page with the data you need, and in the python script you get raw data, without executing JS.
Read about selenium, it will help in such cases. Or study what requests the site sends and simulate them.

Answer 2

This is the ID for which the price is then set. The price is uploaded via POST to https://www.dns-shop.ru/ajax-state/min-price/
1. Open Ctrl+U and see what is on the page
2. Open F12-Network and see where prices come from and how they appear