B
B
beloys2018-06-01 20:59:43
MySQL
beloys, 2018-06-01 20:59:43

How to organize storage of MySQL big data (30 million-50 million) and selection by date range?

Good day to all!
Faced the problem of storing large data, I'm afraid to mess up, so I need advice from more experienced developers.
The task is as follows, it is necessary to store data on currency quotes over time and make a selection on them over a period of time. I met implementations such that each quote value is stored in a separate table and conditionally linked by INT TIMESTAMP. But here the server returned a different frequency of data depending on the specified period of time, with a slice of up to a week, this is 5 minutes, with a slice of up to a month - 1 hour, how to organize this on the server is a question for me. Which range operator is the most efficient? Which data type is better to use for date INT or DATE in this case?
I haven’t worked with such large data before, as I believe, I won’t be able to rake up with indexes alone, and partitions won’t help at all, because the intervals can be completely different.
Please share your experience / your decisions / poke your nose, tk. at the moment I don't even know in which direction the right decision can be.

Answer the question

In order to leave comments, you need to log in

3 answer(s)
R
Roman Mirilaczvili, 2018-06-02
@beloys


Choosing a DBMS for storing time series

A
Alexander Aksentiev, 2018-06-01
@Sanasol

30-50 million is not big data. And everything will fly.
The usual logic, what's so difficult about checking the size of the selected period? If you have chosen a lot, then we make the grouping more.
timestamp should be faster and more convenient, especially in date selections.
Well, as a bonus, the convenience of working in the form of all sorts of date_sub() date_add(), etc.

T
thyratr0n, 2018-06-02
@thyratr0n

Usually such things are organized in different tables, like: stock_5_min, stock_15_min, stock_1_hour. As a rule, in exchange clients, the time periods are always the same and unchanged.
And at the expense of time periods - shift it to the application, not to the database.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question