B
B
Bjornie2017-10-13 19:18:37
MySQL
Bjornie, 2017-10-13 19:18:37

Where is the best place to save parsed data through Scrapy?

To begin with, I would immediately like to clarify that the final data format of interest to me is .XML. But still, I would like to know where it is best to save data when parsing, so that it is: the fastest, most reliable and convenient.
After reading some topics and lessons, I realized that it is quite enough to store structured data in MySQL, the reverse - in Mongo.
Also, the convenience of the database is that you can check on the fly whether there is already a specific product in the database (in our case, by ASIN), and in this case, either update it, or add a new one, or do nothing.
I will add (if it is important): the parser is planned to be installed on some VPS and about 500k products (prices and availability) will be parsed. speed and stability are very important.
PS I have never dealt with NoSQL.

Answer the question

In order to leave comments, you need to log in

1 answer(s)
L
longclaps, 2017-10-13
@longclaps

Most quickly - in memory, reliably - in a notebook, conveniently - it depends on habits.
All popular DBs are fast enough not to be a bottleneck on this task.
But the parser may well turn out to be one.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question