D
D
Dasha Tsiklauri2018-04-18 18:51:38
big data
Dasha Tsiklauri, 2018-04-18 18:51:38

DB for frequent inserts and large amount of data?

Tell me the engine for storing large volumes of ~ 50GB per day. The data is well structured (suitable for storage in tables, 7-8 fields), index by 1-3 fields (id, date, id_by which_we aggregate).
There are about 40 billion records that are supposed to be in the database in their original form (3 fields), by some request a subset will be selected and placed in the cache. Requests about 20-30k per second. Will the initial data be supplemented with several columns (dates, numbers) and fit into another database or table / collection, or will the original data be modified?
First of all, I am interested in a solution that is productive for insertion, which is easy to scale (for a smaller number of actions, add +1 node to the cluster). Secondly, I would like to see examples for Docker Swarm.
Thank you.
UPD:example
Initial table 40billion ~ 400 collections of 100k records of type PrimaryData (if in terms of nosql), all collections are identical in structure.
Data is pulled out from the source, supplemented, we get ExtendedData and inserted into another database. Search by second base or by key (id) or by date range with aggregation by DataAgg field

type PrimaryData struct {
  ID    string `bson:"_id"`  //char(16)
  Data1 int32  `bson:"d1"`
  Data2 int64  `bson:"d2"`
}

type ExtendedData struct {
  ID      string `bson:"_id"` //char(16)
  Data1   int32  `bson:"d1"`
  Data2   int64  `bson:"d2"`
  Data1TS int64  `bson:"d1_ts"`
  Data2TS int64  `bson:"d2_ts"`
  Data3   int64  `bson:"d3"`
  Data4   byte   `bson:d4`
  DataAgg int32  `bson:dagg`
}

Answer the question

In order to leave comments, you need to log in

1 answer(s)
X
xmoonlight, 2018-04-18
@xmoonlight

TimescaleDB

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question