How can you store a list in memory in Python so that it eats less than this very memory

Question

Python

There is a list or a tuple in it to store strings from 10 to 13 characters long. There are several million such lines.
Such a list weighs quite a lot. If it were an array like in C, this array would probably take up less memory. Is it possible somehow in python to pack this list in a similar way.
The list already has a known length, there is no need to resize. The only thing needed is to be able to run through this structure using for in or any other iteration method.

And if you know a similar solution for dict, I'm also interested.

Answer 1

Maybe, if you are not afraid of C, and you are sure that it will turn out more compact, you can use it through ctypes , for example? With the support of such a solution on * nix systems, there will be no problems.
Another perverted way that came to mind is to mount a ramdisk using fusepy , create a file in the ramdisk using zlib and seek on it. :)
Well, the third option is to think again. Are your strings unicode or ASCII? Is it necessary to keep the entire list in memory (is this a performance requirement?) or is it acceptable to load the next portion, for example, in the background, when the current one comes to an end?

Answer 2

My test showed that 1 million lines (10-13 characters) in a python list takes up about 40 MB in memory. Which is a bit much, but not too bad.
Tested ctypes like this:
>>> from ctypes import *
>>> FixedSizeStr = c_char*13
>>> MillionStrings = FixedSizeStr*1000000
>>> ms = MillionStrings( *(FixedSizeStr(*str(i)) for i in xrange( 1000000)))
This is +215 MB?!..
The variant proposed above with encoding to a number, although it saves space, will significantly eat up performance if you run through this list and compare with another string.
So if you really need optimization, you will have to write your class in C / C ++ and export it to python. Everything else will be half measure.

Answer 3

Use sqlite database. It eats little memory, it works fast (of course, provided that you opened it in memory by calling sqlite3.connect(":memory:"), and not in a file, but that's what you need).