Answer the question
In order to leave comments, you need to log in
Where is the best place to save parsed data through Scrapy?
To begin with, I would immediately like to clarify that the final data format of interest to me is .XML. But still, I would like to know where it is best to save data when parsing, so that it is: the fastest, most reliable and convenient.
After reading some topics and lessons, I realized that it is quite enough to store structured data in MySQL, the reverse - in Mongo.
Also, the convenience of the database is that you can check on the fly whether there is already a specific product in the database (in our case, by ASIN), and in this case, either update it, or add a new one, or do nothing.
I will add (if it is important): the parser is planned to be installed on some VPS and about 500k products (prices and availability) will be parsed. speed and stability are very important.
PS I have never dealt with NoSQL.
Answer the question
In order to leave comments, you need to log in
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question