How to store a huge array of objects in a file?

Question

.NET

Assume that the array contains a set (from 10,000 to 100,000) of objects of various classes with several types of properties and variables. All object classes inherit from a base class. All values ​​of variables and properties of such a set of objects are unique. You need to store and read object data from a file.
It is not at all suitable to store objects in a format, say, XML or JSON - the file size is too large, which eats up space and transfers figs over the network. In what form to store the data of these objects with minimal disk space?

Answer 1

Assign an id (an integer) to each class, and write functions for writing to a binary stream and reading from it for this class. Record field values ​​directly.
If direct access to objects is required, then when creating a file, allocate at the beginning a place for references to the position of each element of the array in the file, then, as the file is created, accumulate these references. Write down at the end. If direct access is not needed, write and then read all objects in a row.
If the class structure will change, and sometimes you need to read old files, you need to provide a version number so that the class deserializer can take it into account. It will not work to read the new file with the old program.
If there is information about the ranges of field values, you can write fewer bytes than the size of the type (for example, for integers - two or three bytes). For arrays, packings can be provided by some difference schemes.

Answer 2

Storing structured objects in a file is not a good idea, especially at this size. If it is not possible to use MySQL or other service databases, maybe you should look towards sqlite? If you need it in a file, then you should turn to the capabilities of the file system - these are SSD drives, RAID1 (RAID2, if there are at least 4 hard drives) or a RAM disk (in the latter case, it will take care of backup, and even better - mirroring to a real hard ).

Answer 3

I support Mrrl file after saving\serialization, you can easily compress zip, if there is no jpg inside, it will compress every 2-20 times

Answer 4

Serialize to XML and compress with gzip.