Answer the question
In order to leave comments, you need to log in
What is the best way to store intermediate parser data? NoSQL or files?
Good afternoon!
I want to say right away that the question is probably stupid, since I have never worked with noSQL and asked this question only because NoSQL is now an uptrend, which is difficult for a developer to ignore.
I have an idea to write a small parser for myself. The work of the parser will be reduced to viewing each page of the specified site and collecting some statistical information about it: title, header, and so on.
What is better to use to store the intermediate results of the parser? This is the data that is obtained after each page. Now I do everything on files, but I began to think about the correctness of this approach.
While the parser is running, only data is written, reading is minimal. In the future, simultaneous parsing of several sites by several people is possible.
Will there be any gain (in speed or in reducing the load on the server) if I transfer the work to noSQL?
Answer the question
In order to leave comments, you need to log in
it doesn’t matter at all where to put this data, you can also put it in a mysql relational database, you can also in nosql, the main thing is that the data in the database is where you can quickly find and retrieve it.
And if every time after parsing a page to do an INSERT in MySQL, the load will not be very large when processing 2000 pages?
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question