How to track xhr requests?

Question

Parsing

Hello! I became interested in the topic of parsing and do more or less simple things using the curl or casperjs tools. Difficulties are caused by sites that are formed dynamically. For example, there is a site that is generated through ajax requests. When studying it with the help of the chrome debugger, I found that every time the "more" button is pressed, 20 product names are loaded on the page, an xhr type entry appears in the debugger each time - apparently this is xmlHttpRequest. Examining the headers didn't tell me anything - they are all the same, the only difference is in the size of kilobytes and there is also form data, in which there is a page variable with the page number. Share your experience on how to use it, how to identify a request, and is it possible to somehow send it and receive data using curl? Is there any method or tool

Answer 1

I assume that xhr sends a request like domain.com/article?page=2, etc. and at the same time it is known that there are 20 records on 1 page, it remains to learn how to determine which page is the last one. Unless, of course, there is no hint anywhere how many total pages or total records.

Answer 2

google: curl xhr example
Well, of course you can use curl to form a post request with a page number and request data. You can also add an xhr header.
"X-Requested-With" => "XMLHttpRequest",

Answer 3

Scraping an infinitely scrolling page
https://habrahabr.ru/post/312816/