M
M
maxcad2021-09-21 22:49:41
C++ / C#
maxcad, 2021-09-21 22:49:41

Does the program hang when writing a lot of small files to the disk?

Question. How to correctly organize the recording of tens of thousands of small files to disk?
Up to 3000-5000 files of 40Kb are written quite quickly. And if there are 200,000 or a million of them, then the program freezes and the disk stops responding and after a while the system hangs and you can restart the computer only through the reset button.

The entry code in a simplified version looks something like this.
There is an array files[] with file names. We go around it and write it down.

foreach(string filename in files)
 {     
       File.Copy("src.txt", filename, true);
 }


When the program and the system hang, after a reboot, there are approximately 20,000 files left. The cycle is designed to record approximately 100,000

Answer the question

In order to leave comments, you need to log in

2 answer(s)
M
maxcad, 2021-09-22
@maxcad

You can delete a topic.
The recording was made on a disk encrypted with VeraCrypt. On regular discs it is written without problems.
VeraCrypt hangs, this is her problem. Multiple recording of small files is like that.
Here the campaign needs to pause so that the encrypted container can catch its breath.

V
Vasily Bannikov, 2021-09-21
@vabka

1. Use File.OpenRead, File.OpenWrite, and WriteAllBytesAsync so that the processor is not idle while files are copied
2. Copy them in parallel, not one at a time.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question