How to read a large file line by line?

Question

C++ / C#

I have already seen similar questions on Habré, but when it comes to text files weighing from 40 MB to 5 GB, some value is lost.

I'm trying to load text line by line into a RichTextBox, after processing each line with a certain condition, from huge files with logs. And if the program does an excellent job with a file up to 10 MB, then what to do with 40 GB logs? How, for example, quickly opens such FAR files?

Part of my code:

using(var reader = new StreamReader(new BufferedStream(File.OpenRead(file), 1024*1024)))
{
string line;
while ((line = reader.ReadLine()) != null)
{
// обработка и вывод строки
}
}

- at first I tried to load the entire log into the program and then work with it, but there is so much data that it will simply choke
- now I'm reading line by line, processing and outputting, but it will still choke

How to speed up all this?

Answer 1

Well, as an option, load it in parts: load several dozen lines into memory, process it, throw it away, load the next ones ... You can load the next ones while processing the previous ones, just be careful not to get out of memory.

Answer 2

It seems to be done like this: memory-mapped-files