.NET eats up memory unnecessarily?

Question

.NET

Bonsuar, gentlemen!
In general, I have a program that:

1. Reads a file and writes bytes to the buffer
   
2. Through GZipStream compresses data and again writes them again to the same buffer
   
3. Reads a buffer and writes data to memory
   

Ie look.
The first snapshot is an empty buffer.
The second snapshot is creating a MemoryStream, writing the buffer data ( GC ) to it, using this MemoryStream in a GZipStream , writing the data to the buffer. (GC for both Streams)
The third snapshot is to write the buffer data to a file.
GC - with this I marked the moments at which garbage collection should be.
To be more precise, these moments look like this:

byte[] buffer = new byte[1000000000]; // RAM заюзалась
buffer = null;
buffer = new byte[100]; // (<b>GC</b>) тот же самый буфер. Но памяти при этом выделено под 1000000100 байт.

In the end, I have 0.5GB of RAM to work with a 100MB file.
Read file +100mb
Create MemoryStream +0mb = 100mb
Compress data with GZip and write to MemoryStream +131mb ( compressed data weighs more than WTF????? ) = 230mb
.... +100mb = 330mb
Create file, write to it data from the buffer +0mb = 230
In the end, the GC works on the last line of code -200mb = 130mb (remains from the buffer) I
use each resource through using
Before each recording of a new portion of data into the buffer, I null it.
How to make GC work more aggressively?
Addendum: Why the hell is a GZipStream with CompressionLevel.Optimal producing more data than it was?(not relevant for really compressible data like test files, images)

Answer 1

It's hard to say anything without seeing the whole code. But here:

byte[] buffer = new byte[1000000000]; // RAM заюзалась
buffer = null;
buffer = new byte[100]; // (<b>GC</b>) тот же самый буфер. Но памяти при этом выделено под 1000000100 байт.

There is no guarantee that the garbage collector will work. From can work, and maybe someday later.
What is happening in this code? You allocate memory, buffer = null does not mean that the memory should be freed, but just that the variable refers to another memory location (with address 0). Then you allocate more memory and now the variable refers to a new piece of memory ..
About the garbage collector. In 99.999% of cases, when a programmer wants to work manually with the garbage collector, they need to edit the code.

Answer 2

1) compression algorithms use additional. memory for generating dictionaries. The size of the dictionary will depend on both the compression algorithm and the data to be compressed.
2) in order to free memory from a heap of junk - you can do GC.Collect to initialize the garbage collector. In this case, you have to do GC.Collect(2), which is quite an expensive operation.
in addition, you should be aware of the nuances of garbage collection for LOH. Unlike SOH, LOH will not be defragmented in memory, and the CLR will simply try to reuse the freed chunks. Therefore, it is quite possible that there is memory on the heap, but the CLR continues to allocate it for new objects.

Answer 3

The collector will give memory when the program is idle and / or the system needs it (memory) and / or less than some% of the memory is available in the system.
According to the GC call schedule, it was called only at the very beginning.
If you want a different behavior, move the selection and use of the array into a function, and loop the function, while setting gc to have a parallel server assembly. Then the discarded stack frames with a pointer to your old byte[] should be quickly picked up by parallel garbage collection, the application will begin to give away memory more intensively.
Relevant flags in appConfig
https://msdn.microsoft.com/ru-ru/library/ms229357(...
https://msdn.microsoft.com/ru-ru/library/ms229357(...