How to serialize / deserialize a 1gb list object as quickly as possible in c#?

Question

C++ / C#

I'm doing the deserialization now

BinaryFormatter bin = new BinaryFormatter();
var test = (List<abc>)bin.Deserialize(stream);

But it is very long.
How to do faster?

Answer 1

So maybe better in pieces, and then just glue the pieces together? Why the whole list at once? There is clearly a problem with the problem statement.

Answer 2

Depends on what kind of objects you have there.
Under certain conditions, you can significantly speed up if you read the stream using BinaryReader and create objects by hand.
Try it in general.

Answer 3

the overhead of binary serialization is very small, if in general the download speed of such an object is equal to the download speed of the same 1 GB of data into the RAM, here the loss is not on the algorithm but on the speed of the HDD, I would advise you to disassemble the task into parts, I don’t think that the entire list should be loaded into memory.
It is necessary to revise the algorithm, as an option, if in each object of the list there is a certain object that occupies most of the memory, it can be stored separately, and in the object when it is loaded from a file if necessary. Then the list itself will most likely be small (logically, it can be assumed that no more than 50 mb) and the files will be uploaded as needed.
If the object does not have an element that consumes a lot of memory, but there are so many objects in the list, then either the client-server architecture or the database (sql for example) should be used.