N
N
no_one_safe2016-03-10 10:15:40
MySQL
no_one_safe, 2016-03-10 10:15:40

How to organize the storage of large amounts of data?

Good afternoon.
I want to consult with the community on the following issue:
Essence:
There is a project on the first ZendFramework (I think that this is not so important)
There is a table in the database of the form:
643fcd48188b4df8a265672971672c06.png
As you can see from the screenshot, the table has the following structure:

idPRIMARY AUTO_INCREMENT
nameEntity name. Can be repeated (different id with the same name have different characteristics) - never changes
lowA certain characteristic - its lower indicator - a variable value
middleThe same characteristic - its average indicator - a variable value
highThe same characteristic - its high indicator - a variable value

Records in the table of the order ~30K. About 2000 records are added every year .
Every day , approximately ~15k records change low, middle, and high values.
On the site, the entity page is the output of all entities with characteristics grouped by the name field.
Task:
I would like to collect statistics on the change in low, middle and high values ​​(daily change for about 15,000 records) for at least a year and beautifully show everything on graphs on entity pages.
Question:
Which data storage strategy should I choose?
Obvious solutions:
  • MySql - create a table with id, entity_id, low, middle, high, date - add 15000 records there every day. For the year -> 5475000 entries. Not really something I like it. Or the norm?
  • File storage - we create 15000 files, where we store data in a serialized array. Every day we change all 15,000 files. It's kind of crutch too.
  • I am also considering options to somehow optimize storage by combining data by the name field. So far, nothing good has come to mind.

The entity storage table itself cannot be changed (only fields can be added)
Any ideas?
UPD:
Thank you all for your opinions. I will stop all the same on the first option with the MySql table. I was frightened by the figure of 5 million, which was somewhat reckless.

Answer the question

In order to leave comments, you need to log in

2 answer(s)
M
Melkij, 2016-03-10
@no_one_safe

5.5 lyam records per year ?
For ten years, ridiculous 55 lyams? Right, what a bullshit. Even for mysql. What makes you think that this is big data? Here are 50 lyams daily - you can already call it big data with a stretch.
Two tables:
id name and what else is needed for the entity itself.
date, entity_id, low, middle, high. You can partition quarterly.

S
sim3x, 2016-03-10
@sim3x

Don't do premature optimization
For starters, 3NF storage is enough

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question