Scrapy how to save large amounts of data?

Question

Python

Hello everyone and thanks in advance for your replies.
Do not kick, just started to figure it out, tell me how to save large amounts of data? If all this in csv to unload more than 400,000 lines, will there be problems later?

Answer 1

If all this in csv to unload more than 400,000 lines, will there be problems later?

And what does Scrapy have to do with it?????
You know better what you are going to do with this file.
With sequential line-by-line processing, there will be no problems even with billions of lines.
But if you need to delete arbitrary lines or process them not in a row, this is more gemoine.

Answer 2

ok, thank you all, I will upload everything to the database