Is NodeJS good for web scraping?

Question

HTML

Recently, at work, I often have to parse sites. Now I mainly use PHP Simple DOM HTML Parser, but I want something more advanced - for example, to be able to automatically follow links in pagination, etc. I'm looking at NodeJS and its modules. Is it worth delving into this topic or is this infrastructure not much different in capabilities from PHP-based solutions?

Answer 1

great, as mentioned above - phantomjs is very cool!

Answer 2

A good alternative to phantom is Nightmare: https://github.com/segmentio/nightmare

Answer 3

It's not about the language. And in the instrument.
For example Scrapy is a great tool. It's in Python. But I think there are also PHP and JS (Node) you just need to look for.
If we are talking about full JS emulation, then yes, only JS.
But it's not NodeJS - it's essentially slash JS.
A headless-browser in JS - this provides full emulation of the browser DOM and full control. For example PhantomJS. But if you don't need full DOM emulation to parse your site (the site isn't a fancy AJAX site), then you don't need PhantomJS.
Then just look for a convenient library for your favorite language...

Answer 4

https://ru.wikibooks.org/wiki/Grab

Answer 5

When you have a crowbar in your hands , everything seems to be a site
Worth a look at golang

Answer 6

Никто не пишет про cheerio.
Неужели он на столько плох? Мне показалось, что он достаточно облегчен и алгоритм прост (загрузка-расчленение-сохранение). Возникла только проблема с циклом. Когда прогоняешь его по всем страницам в цикле - он пытается этот цикл выполнить одновременно(синхронно). А вот как заставить цикл продолжать выполнение только после завершения предыдущего шага - не понял пока. Если можете - ткните носом, пожалуйста!
Идея была написать свой API к сайтам, которые этого не предусматривают...

Answer 7

Посмотри в сторону ANTLR. Сам инструмент на Java, но есть targer в JS

Answer 8

I would advise python or go, parse with nodges, also with fanotomzhs - you will need a cloud of iron resources