What tools are more convenient to process text files?

Question

csv

Prompt tools for processing text files / scripting languages ​​that contain convenient mechanisms for this.
You need something like a DBMS for text files.
Files - downloads from various systems (very different). There may be many. Volumes up to 10 GB, several tens of millions of lines.
It is necessary to: aggregate data from uploads / change according to a certain rule, select records that meet the criteria / statistics based on data / summary data.
Tool requirements: Easy to install and carry, high speed and flexibility.
Desirable: regular season, low entry threshold for users.
Platform: Windows 7 x64.
For hardware: i5-2520M, SSD, 8 GB RAM
Paid / free - it doesn't matter.
Why not classic databases: it is not always known what type of data is contained in the fields, some records may be broken. In principle, indexes are not needed - 10-15 reports on the provided data are required.
But if you tell me something that still fits - I will be glad)

Answer 1

Lua or Python. LuaJIT is very fast, there are patterns instead of regular expressions that are also fast.
As I understand it, there is an aggregation from many text files of different formats into one? Let's say CSV or line-by-line JSON.