E
E
Emil Revencu2015-06-03 23:08:47
Python
Emil Revencu, 2015-06-03 23:08:47

What language to use for a dynamic site parser?

There is a site with data that is loaded when the page is scrolled or when buttons like "show more" are pressed
Loading via Ajax, but the received data is processed by scripts and displayed according to a certain algorithm.
How can such content be parsed?
Tried through VBS. An excellent solution with imitations of button presses and page scrolling in InternetExplorer.Application.
But this is only on Windows. How can such tasks be solved in JavaScript or Nodejs.
If possible - give examples
Yes! And further! Sites are https!

Answer the question

In order to leave comments, you need to log in

3 answer(s)
A
Alexey Rytikov, 2015-06-03
@chlp

Maybe Selenium will work?
www.seleniumhq.org
habrahabr.ru/post/186496

K
klinkin, 2015-06-04
@klinkin

I used phantomjs.org

A
Alexander Pinkevich, 2015-06-04
@pinkevich

If I understand the question correctly, then you need to find the url to which the ajax request is sent on the site (F10 > Network) that loads the content, and send the same post ajax to it only with your data (page=1, page=2 or offset =20, offset=40...)

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question