What is the optimal buffer size?

Question

.NET

Good time of the day. Are there any articles on choosing the optimal buffer size for reading a file and writing to a file?
On the Internet, I just came across this article from 2004, where it is recommended to use 4 kb or 64 kb, but because 13 years have passed and the power of computers has become greater, then there is little trust in such articles. On the stack, people also write that they say 4 kb is better to use.
It's just that my work will most often be with files of more than 10 GB and 4 kb for them to take it somehow frivolously (long work with the file)
PS this is necessary for a diploma.

Answer 1

Well, since this is necessary for a diploma, it would be quite appropriate to carry out an analysis: a series of tests for reading / writing large files, say, 10GB in size.
Read / write different blocks starting from 512 bytes, then a power of two (1024,2048,4096,8192, ...) while measuring the speed ... the program is simple, I think you can manage it in an evening ... build a graph of performance versus buffer size ... there will be a chic section of the diploma ...

Answer 2

In fact, the ideal buffer depends on a lot: on the block size of your file system, on the cache, and so on. To summarize, the main thing is not to exceed the area of ​​​​processor caches, but registers in such large sizes are out of the question, of course. So you can safely take exactly as much as L1 cache will fit. Until you climb higher, you won’t get into the RAM area, and therefore you won’t go down the performance curve.
I would advise you to batch a 64kb file and not to carry out early optimization , write it as is first, and then deal with such micro-optimization. I don't think there will be a big expansion in terms of performance.

Answer 3

I looked at the Microsoft sources, and there, in relation to C#, buffer 81920 is selected for copying files, this is explained as follows: "We select a value that is the largest multiple of 4096, but which is still less than the large object heap threshold (85K)." I think that this is the most optimal buffer for copying.