How to do complex page parsing?

Question

Parsing

Thank you for stopping by ...
It's not an easy (for me) task. How to get the code of the page that is located by the URL if:

1. browser emulation is needed, the page is protected by some algorithm that distinguishes the browser from the user
   
2. you need to parse html that is loaded dynamically after filling out the form
   

That is, the algorithm is as follows: browser emulation, filling in form fields, waiting for a response, data parsing.
Who met with this task at least partially, please give a tip or an example on what to write, where to dig. Language and platform is not important. Thank you.
ps The task is abstract... I set myself the task of writing a parser. Difficult (for me, at least). For example, parse search results for aviasales or get data after ebay authorization.

Answer 1

Do you have a single programmer in your web studio?
The hint is simple - look at the AJAX request with the inspector, copy it for cURL (right-click on the request itself, copy as cURL) and repeat, in fact, with the curl. If the answer is the same, then you have already solved the problem and it remains only to parse the data.