How to parse a dynamic page?

Question

Python

So, we have a patient:
the VTB website
understands that coins are generated automatically, so we climb into the developer tools.
parsing network (xhr) led us to this link:
https://www.vtb.ru/api/sitecore/coinsapi/direct?co...
We study further and get:
https://www.vtb.ru/api/ sitecore/coinsapi/filter
We look at the headers of this link:
- request method - POST
- response - JSON
inside JSON is (url: search_url)
it is clear that search_url needs to be substituted in https://www.vtb.ru/api/sitecore/ coinsapi/direct?co...
and get a static coin page.

problem:

import requests as req
json_url = 'https://www.vtb.ru/api/sitecore/coinsapi/filter'
headers = {'user-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:72.0) Gecko/20100101 Firefox/72.0'}
resp_json = req.get(json_url, headers = headers)
print(resp_json)

and the answer is clear: 404 (page does not exist)

the question is: how to correctly send a POST to such a site?

Answer 1

Well, for starters, user-agent is optional and Content-Type is required if you plan on getting json. Also, in the post request, json is passed as a parameter, so we also need to pass it.

import requests

headers = {'Content-Type':'application/json;charset=utf-8'}
url = 'https://www.vtb.ru/api/sitecore/coinsapi/filter'
data = '{"query":"","newCollection":false,"Discounted":false,"GiftBox":false,"Order":"priceAsc","SearchGroups":false,"CoinList":"all","Favorites":[],"Groups":["8e67bb77202c40fa8ae0258d5bcb66f8","087e7eca08724e88aa1fbd0ebb0ebf70"],"Series":[],"Themes":[],"Metals":[],"PriceMin":"","PriceMax":"","Page":1,"ResultsOnPage":16}'

response = requests.post(url,data=data,headers=headers)

In json, which is in data, you can change various parameters, such as: page number, number of coins per page, minimum and maximum price, etc.
After these manipulations, we get "200 OK" and json with coins.