How to reduce memory consumption when running python scripts?

Question

Python

Hello! I wrote a parser in python, 6-7 scripts work daily at the same time. In this case, several pythonw.exe processes hang in the task manager, 3-4 of which immediately after launch take 1.5 GB of RAM each. But at the same time, I noticed that after a few hours, only 1 or 2 processes consumed 1.5 GB each, and the rest were 10 times less. How to make it so that during startup and further work the consumed memory of pythons is not large?

Answer 1

How to make it so that during startup and further work the consumed memory of pythons is not large?

Without seeing the code, it's hard to judge anything, but I can give advice. When parsing, when you extract the characteristics of a "unit" of data (an online store product or a tweet on Twitter), instead of passing a list containing many "units" of data to another function, pass them one by one using yield.

Answer 2

use not a DOM parser - it puts everything in memory, but a SAX parser, for example lxml.de