How to optimize write speed of many large files in linux?

Question

linux

There is a need to write data in parallel to many (~100) files, the resulting size of which will be quite large, but approximately known. How can you tell the kernel what the size of the resulting files is expected to be, so that you don't have to do unnecessary file moves on disk as they grow?
Thanks in advance for replies.

Answer 1

Using fallocate to create a file of the required size, if the file system does not support sparse files (leaky files that do not take up disk space), then this operation is instantaneous and the file will be filled with garbage (data that lay in sectors).
Unfortunately (or rather fortunately, because security is more important) all modern file systems support holes (they will return zeros), i.e. physically the file won't take up any space, and you will either need to fill the file with zeros yourself or use the fallocate - FALLOC_FL_ZERO_RANGE option, this physically writes the data to disk, so the operation wastes time.
ps file systems with compression or online deduplication (such as zfs) will not allow you to control the creation process, more precisely, this will need to be done by file system utilities.
upd. I forgot to remind you that on Copy On Write file systems (such as zfs / btrfs / ...) preliminary file allocation is pointless, even harmful, since any write will go to a new location on the disk.

Answer 2

What is "moving files on disk"? Do you mean defragmentation?
If it is critical to have a file in the form of a continuous (as much as possible) area on the disk - wait until the recording is over and copy the data to an idle device, everything will be beautiful. Well, or allocate space on the main disk immediately, and not as the files grow.