Answer the question
In order to leave comments, you need to log in
How to optimize data storage?
Greetings.
Having not achieved sane behavior from Mongo, it was decided to migrate to postgre.
The essence of the project is the collection of information from various sources, partly parsing, partly the users themselves, partly with the help of human resources. The volume of "records" at the initial level is expected to be ~50 million. Each data group has 5 to 10 fields.
Hence the question of how best to build a data storage structure, so far there are only two options in my head.
1. Store everything in one large table of the form:
ID / уникальное_поле / группа_полей_1 / группа_полей_2 / группа_полей_3
- ID / уникальное_поле / данные_для_выборок
- ID / уникальное_поле / группа_полей_1
- ID / уникальное_поле / группа_полей_2
- ID / уникальное_поле / группа_полей_3
- уникальное_поле = aaaa
- группа_1_поле_A = xxx
- группа_2_поле_A = yyy
- группа_3 - нет записи вообще
- уникальное_поле = aaa, группа_2_поле_A = yyy
- ID / уникальное_поле / итоговая_группа_полей
- ID / уникальное_поле / группа_полей_1
- ID / уникальное_поле / группа_полей_2
- ID / уникальное_поле / группа_полей_3
Answer the question
In order to leave comments, you need to log in
In general, through the spent nerves and trials, we revised the scheme of working with the base. At some points it got worse, at others it got better. But as a result, all the data was split into several tables.
- one common with the "final" data and all reading selections work on it
- three tables of intermediate data with which they work for editing and some specific selections
- one service format table, in principle, only automation works with it
- well, a little triggers and stored ones, which actually implement the atomic logic of working with data
In principle, it turned out optimally both in terms of working with data and in terms of resource intensity ... I consider the question closed =)
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question