For a parser with support for JavaScript, cookies, what bundle would you recommend?

Question

JavaScript

It is required to write a parser for a complex site, with authorization, checking cookies, user-agent, captcha. Some things are generated by the script.
I read on the Internet, people are increasingly using nodejs for this, as I understand it, in conjunction with NW.js.
What else is required?
What other options are available?

Answer 1

Python + Selenium, use PhantomJS web driver in Selenium.
For cookies I use pickle.

Answer 2

for such things, as a rule, 2 applications are written:
1) scripts for phantomJS
2) http-api (runs locally with a phantom) to receive parsed information in a structured form from a phantom. For my part, I recommend .net for the implementation of this part (the application is done in minutes, all the conveniences of linq, etc.).
* to scale the first module, you can write a simple parsing process manager that runs as many instances as you need on the right hosts with the right configuration (on the same .net)

Answer 3

Yes, it is possible to implement this on a regular request quite calmly, there are cookies, and whatever headers you like.
You need to look, sometimes it's easier to figure out how the necessary requests are generated, but if they are generated by a cunning javascript, you can always download this javascript, put it in vm and also generate them.
Well, or as they wrote here, headless browsers.
And yes, the node is ideal for interacting with third-party sites.

Answer 4

In scrapy, all this can be stuffed

Answer 5

https://github.com/puppeteer/puppeteer

Answer 6

How to make such slider(?) of images as on the image? - see something from here

Answer 7

any - from you it is necessary to adjust styles only.
in general, I recently really liked the multifunctional and lightweight here:
Sly.js

Answer 8

codecanyon.net/item/boutique-carousel/full_screen_...
codecanyon.net/item/jquery-carousel-evolution/full...
here is something similar