Answer the question
In order to leave comments, you need to log in
Which DBMS and database structure to choose for storing ~160 billion records?
Essence of the project
About 100 wells are being drilled, data is sent from them to the central server every second .
Wells are drilled on average 2 months.
Data of this type:
Answer the question
In order to leave comments, you need to log in
For storage - key-value, for OLAP - column database. For graphs - and dependence on realtime. Or stupidly streaming from ingest or regular aggregates to another database.
The muscle will do just fine.
160 billion records of 10 bytes is ~ 1.6 TB, not so much.
Charts are displayed not from the main database, but from the slave replica (to reduce the load on the main database).
Most likely, graphs can be displayed with less accuracy: take numbers not once per second, but once every 10 minutes. Because the numbers "once per second" will still not be visible. Those. for graphs, you can make a separate, much smaller database.
You can also think about how often the parameters change? If the parameter hardly changes, maybe record it at longer intervals? For example, once a minute? And in case of some abrupt (catastrophic?) changes, resume recording once a second.
Log-files can handle storage... Exactly like MySql, mariadb or postgresql ... or KV ../ yes, you can store it in any way you like.
Question: what to do with this saved data and how to process it. And depending on the hatelok, the architecture will emerge and you can choose a base suitable for the task.
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question