S
S
Sergey2019-12-15 16:27:12
Database
Sergey, 2019-12-15 16:27:12

Which DBMS and database structure to choose for storing ~160 billion records?

Essence of the project
About 100 wells are being drilled, data is sent from them to the central server every second .
Wells are drilled on average 2 months.
Data of this type:

  • Time
  • Depth
  • about 300 parameters

The parameter looks like this:
  • Parameter ID: string(3)
  • Value: float

If I didn’t mess up anything, then about 1.6 billion lines will be used for one well (for storing parameters), for all wells - 160 billion lines .
It is required to organize the system on this server so that it accepts and stores these parameters.
And also displayed curves of changes in parameters in real time. Plus curves for the past with a change in the scale and viewing range.
The current program uses MySQL, with separate tables for each well. The largest table occupies 30GB and 160 million rows (the current program saves fewer parameters).
Now questions:
  • Is PostgresQL sufficient for these purposes with 2 tables?
  • Or does it make sense to create separate tables for each well?
  • Maybe you should use NoSQL?
  • Maybe this applies to Big Data and appropriate solutions are needed?

Answer the question

In order to leave comments, you need to log in

3 answer(s)
I
Ivan Shumov, 2019-12-15
@inoise

For storage - key-value, for OLAP - column database. For graphs - and dependence on realtime. Or stupidly streaming from ingest or regular aggregates to another database.

T
tester12, 2019-12-15
@tester12

The muscle will do just fine.
160 billion records of 10 bytes is ~ 1.6 TB, not so much.
Charts are displayed not from the main database, but from the slave replica (to reduce the load on the main database).
Most likely, graphs can be displayed with less accuracy: take numbers not once per second, but once every 10 minutes. Because the numbers "once per second" will still not be visible. Those. for graphs, you can make a separate, much smaller database.
You can also think about how often the parameters change? If the parameter hardly changes, maybe record it at longer intervals? For example, once a minute? And in case of some abrupt (catastrophic?) changes, resume recording once a second.

S
Sergey Eremin, 2019-12-16
@Sergei_Erjemin

Log-files can handle storage... Exactly like MySql, mariadb or postgresql ... or KV ../ yes, you can store it in any way you like.
Question: what to do with this saved data and how to process it. And depending on the hatelok, the architecture will emerge and you can choose a base suitable for the task.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question