How can I insert data into a file in C++ or C without loading the entire file into a buffer?

Question

C++ / C#

Hello. Faced the problem of handling very large binary files. It is highly desirable not to place them completely in the computer's virtual memory. To search for them, a header with sizes located at the beginning of the file is used. If you do not delve into the question, then you need a non-resource-intensive way to do this:

"Hello world" => "Hello other world"

That is, add data to the file without overwriting everything that comes after the "cursor" (ofstream, FILE does not matter). I already tried the method with an intermediate buffer, in which I put a certain amount of file data after the editing place and, to make a shift in the cycle, overwrote all the data going further through this same buffer. Of course, the speed is frankly lame, starting from 800 MB.

Answer 1

Only one: to find a solution that does not require writing arbitrary text at the beginning of a huge binary file, shifting all other data.
For example, it may turn out that in this place you can add a line of 100500 spaces in advance (how many you need to insert the maximum there) and you just write the necessary one over them. One block will be overwritten, the rest will not need to be driven across the disk.

Answer 2

There is no such way. It is impossible to embed a certain number of bytes somewhere in the middle of the file without inventing your own file system.
The only exception would be embedding a number of bytes into the file that is a multiple of the size of the FS cluster, starting from an offset that is also a multiple of this size, by manually shaking $MFT or an analogue
. But this is dangerous and absolutely technically illiterate.
And the moral is simple - store the data in the DBMS, they are specifically arranged in such a way that this issue is resolved in a non-resource-intensive way.