HTTPSConnection.getresponse() ...reesponse.read() is leaking memory. Not?

Question

Python

Hey!
I had a need to poll a massively remote server with a python.
It turned out that by about 5-6 thousand requests, 2GB of RAM ends.
After a sleepless day, I found the point at which the python “drip”:

on the 32nd line, after the response.read() is triggered, a quarter of a megabyte immediately flies away. And in the future, no del -s and gc.collect-s don't save. Tell me - what's wrong? The server responds with only 2713 bytes of XML, but every thousand requests eats up 250 Mb. This happens regardless of the rest of the operation, parseString does not affect. Although I start a survey inside the worker in parallel with other requests in batches of 20-100 pieces, can this leak? The wrapper for the worker is the most standard one - from the official Python docks. The server responds, everything is fine, everything is stable, but the memory flows exactly after response.read() and always exactly 250 Mb per 1000 requests. Where does the problem lie? Please help.

Answer 1

I would check pdbthis code and response.readwould not enter the function. As far as I understand, this is called HTTPConnection.readfrom httplib, but there are several options - it would be nice to understand at least which branch the code goes to.

Answer 2

Are you still using the Connection object somewhere? If not, make it local. Same thing with response. In general, read() cannot leak, everything is primitive there - direct reading from the socket. Flows most likely after parseString.
Can you see how you call the doHTTPSRequest method? And what happens next? Including with the EBayItem object

Answer 3

What python and operating system do you have, by the way?

Answer 4

An idiotic assumption and a solution that nevertheless worked in a c# project (to be honest, I don’t know python and have never worked with it). Also, tons of memory leaked when processing a web request, the garbage collector did not think about cleaning up memory when reading text directly into the processing function.
Saved about the following:
tempVariable=request.ReadFunction();
parseFunction(tempVariable);
tempVariable=null;