N
N
Nikita Roschenko2016-03-15 16:31:46
MySQL
Nikita Roschenko, 2016-03-15 16:31:46

What is the best way to store MySQL, Postrgresql or Mongo logs?

Good afternoon, there is an application that polls the api of several social networks and saves posts in the database, the purpose of the application is to log posts and process them. So at the moment I use MySQL, now ~ 2.869.300 records are generated in the post table per day. And even with the use of indexes, sampling from there according to simple conditions takes some time, which increases every day.
So that's actually the question, I don't need historical data, that is, the maximum lifetime of the record is 24 hours, after this time the record can be deleted.
I see three possible solutions at the moment:
1) Once every 24 hours, clean up the table from old records, but I'm afraid that I will still hit the id auto-increment ceiling, and even if this does not happen, it's still terrible to see such huge id.
2) Use mongodb with its limited collections, but I would not like to use 2 databases in the application. + At the moment, under symfony3, there is no support for mongodb at the doctrine level (maybe I'm wrong, but 2 days of searching did not give a sane result on how to make them friends, and downgrate to symfony2 is an extreme option)
3) The option to use Postgresql with partitioning by the date the post was added and old tables just drop. But there is my subjective minus, if I used to work with mysql and mongodb, then I will sit down for postgress for the first time and I will have to transfer the entire application to it, which will take some time, and the pluses are not yet obvious to me.
In general, who faced similar problems, help me find the best solution, or if someone has an alternative correct option, write. I will be very grateful for any advice, since I am faced with such loads for the first time and I am afraid of making the wrong choice at the beginning, which will then bring pain.

Answer the question

In order to leave comments, you need to log in

4 answer(s)
A
Alexey Ukolov, 2016-03-15
@Avillions

MySQL also has partitioning. This, in my opinion, is the most reasonable solution for such a problem statement.

D
Denis, 2016-03-15
@prototype_denis

Since records are not needed, so overwrite them by date.
Select old data and change old data to new data by updating the date.

T
terrier, 2016-03-15
@terrier

Looks like a task for Logstash

M
mokaton, 2016-06-30
@mokaton

Greetings!
I ran into a similar issue and am also looking for a solution. Have you already stopped at something?
My situation is this - there is a task to store the "logs" of the API for debriefing in case of any problems. And since the web service is already running, I don't want to add an extra layer that would add an increase in response time.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question