How to read large files in QT?

Question

Qt

Greetings.
I rarely write on QT, hence the ignorance of the features.
Faced a problem when reading files larger than 1.5GB.
Suppose there is a 2 GB text file, I want to cram it into memory in the form of a hash table.
As an example:

QString line;

    QFile mfile("bigfile.txt");
    if(!mfile.open(QIODevice::ReadOnly))
           qDebug() << "Error opening file";
    QTextStream in(&mfile);
    QHash<QString, quint64> myHash;
    quint64 c = 0;
    do {
        c++;
        line = in.readLine();
        myHash.insert(line, c);
    } while (!line.isNull());

The fact is that when the application gains about 1300 MB, an error occurs. Apparently, this is OS protection. I would like to hear the correct solution to the problem. The main task is to create something like an array / hash table for convenient work with elements, removing duplicates, sorting, etc. Thanks in advance.

Answer 1

You just ran out of memory.
A 32-bit application on x86 architecture has 2 GB of user memory available by default. Taking into account the fact that the code of the application itself and libraries is rammed into the same memory, and heap fragmentation - numbers of the order of 1.5 GB - the practical ceiling of memory usage.
Or think about how to reduce memory consumption - or switch to 64 bits.

Answer 2

I do not know how in QT, dig in the direction of displaying files in memory.

Answer 3

What exactly do you need to do?
Are you able to process the data as the file is downloaded?
Just don't map the file to memory if it is very large, you can use fopen fread in parts.
I have a suspicion that it's not the file that's the matter, but the banal lack of memory, so just try to compile under x64

Answer 4

The task itself is confusing.
There are no facilities in Qt for working with large files (if I'm not mistaken, there were wrappers over mmap for *nix and its analogue for Win)
In general, you need to map a large file to read memory
Map a file for writing
Read and write from source to destination
Approach quite viable (there are alternatives)
You can look towards HDF5