Primitive C module for Python leaking memory?

Question

linux

Good day! I wrote some utility yesterday using the PyJACK module, and suddenly discovered a strange memory leak, well, it’s hard not to notice it, because in a loop in half a minute all the memory, along with the swap, is ok. The problem is that the Python list that returns the module remains in memory even if some del is called. Please clarify, is it something I don’t know / don’t understand, or is it a bug feature, and how to deal with it in general? Python 2.7, xubuntu 12.04.
So here is a minimal example of the problem.
cmmodule.c

#include <Python.h><br><br>
static PyObject *<br>
test(PyObject *self, PyObject *args)<br>
{<br>
    static PyObject *list;<br>
    list = PyList_New(0);<br>
    PyList_Append(list, Py_BuildValue("s", "nya"));<br>
    return list;<br>
}<br><br>
static PyMethodDef<br>
ModuleMethods[] = {<br>
    {"test", test, METH_VARARGS, "Test for memoty leak."},<br>
    {NULL, NULL, 0, NULL}<br>
};<br><br>
PyMODINIT_FUNC<br>
initcmodule(void)<br>
{<br>
    (void) Py_InitModule("cmodule", ModuleMethods);<br>
}<br>

setup.py

from setuptools import setup, Extension<br>
setup(ext_modules=[Extension('cmodule', ['src/cmodule.c'], include_dirs=['src'])])<br>

And actually run this:
memleak_test.py

import cmodule<br>
while True: cmodule.test()<br>

And I watch the memory flow like a waterfall.
As I see it, is Python's GC undernourished for some reason, or is my code written in such a way that it does not fall into its work schedule?

Answer 1

Everything is in order with the list, the problem is in the interaction of functions. Py_BuildValuereturns a new reference, that is, an object with a reference count set to one that your code now owns. The function does PyList_Appendnot take this object from you, but also increases the reference counter by one more. Now the resulting string will have two owners - the list and your code. When you delete the list, it will honestly take away its one from the reference count, but yours will remain there, so Python will think that someone, somewhere, is still using this line.
You need to or explicitly decrement the counter:

PyObject *list, *item; /* static, право, ни к чему */
list = PyList_New(0);
item = Py_BuildValue("s"; "test");
PyList_Append(list, item);
Py_DECREF(item);
return list;

or use a function that does not increment the counter (steals reference):

PyObject *list;
list = PyList_New(1); /* оставляем место для элемента */
PyList_SET_ITEM(list, Py_BuildValue("s"; "test"));
return list;

(In this case, I used the macro PyList_SET_ITEM, which is just right for populating new lists.)