How to work with tables >1k rows?

Question

Python

I passed the parsing stage, now I need to parse everything. Excel sends me and closes. How to open a table >1k rows(~200MB)? So far, I can only drive it through Python. Pandas do not offer.

Answer 1

Personally, I can use Pandas, but I don't, or very rarely.
Data that is "table-like" in its form is easy to work with. Just what requirements do you make to the methods of working with data. And 200 MB, by the way, is not enough, you can get by with lists.
1. You can pile up "bicycles" through lists and dictionaries. Primitively, a list, where each entry is a dictionary, and the key is the name of the column (field), and then as fantasy tells
2. You can use sqlite, creating a database in memory or on a media. It depends on what kind of python and how much memory, and even from sqlite - sql queries. Several million sqlite easily exports.
Therefore: "bike", sqlite or pandas.

Answer 2

You can look at the visualization using Tableau.
But sooner or later you'll end up with pandas or something like that anyway :)

Answer 3

EmEditor is my ideal. For all types of data, for any task, and of course, much more than 200 MB for a shaman. Open working environment only for 5 gigs permanently.
According to the capabilities of 32 giga files, but in reality, 3-4 gigabytes makes sense through EmEditor, especially if you work with columns and table views.
The rest is easier than grep/cll
No, it's not even close (but for me it's many steps further) than exel. But flexibility, speed and volume, if not the best that they offer, then one of the best.
Of course, you will have to tinker with configs "for yourself" and not 1 week, but it's worth it. While the excel is loading, the task has already been completed in EmEditor) IMHO)