What libraries to use to process large csv files in python?

Question

Python

There are several large csv files, about 1.5-2 gigabytes. These files need to be parsed and processed. What is the best to use? What libraries?

Answer 1

besides pandas, as Homon pointed out, numpy can also be used .
in some cases, working in numpy can be significantly faster than in pandas.
Also keep in mind that dataframes in pandas are, in principle, a slow thing, so if there is a need to often build queries a la sql, then you can raise a local postgresql and perform these operations there.

Answer 2

panda - standard, including all sorts of pivot tables,
but in general I recommend discovering clickhouse + tabix