How to write a modular parser with a frontend?

M

misterkust2014-07-06 08:24:29

Python

misterkust, 2014-07-06 08:24:29

I would like to write a modular server parser that will monitor
the Update of certain sites (1 module - 1 site) and collect the same type of information from them
. Adding it to the database, I think one table is enough. Plus, I would like a convenient web face for users
With flexible access to the collected data.
I am considering mongoDB, python or nodeJS.
What is more suitable and what tools/frameworks would you recommend?

Reply

Answer the question

In order to leave comments, you need to log in

3 answer(s)

L

Lici, 2014-07-06
@Lici

PHP?

E

Evgeny Sofonov, 2014-07-06
@sofcom

If there is no desire to make a bicycle, then - grablib.org - everything is accessible and understandable, much is already in the box and in Python or lxml.cssselect

D

Dmitry Snegirev, 2014-07-06
@Rikkit

In principle, python or nodeJS is not very important. Choose what you like best.
I understand that there should be a site + script that is constantly running and monitoring. It works with the database and, accordingly, it will be managed through the database.
You also need to decide how "realistically" it parses pages. Does JS execute on them (and right now it happens that data is loaded via JS).