In which database to store logs of visits?

Question

MySQL

Hello! I need to keep logs of visits of site users. At the moment I am writing everything to a MySQL table. Once every 5 min. statistics are recalculated. Sooner or later there will be many records, 100 million, 500 million records.
What is the best way to store this information? In which database, so that it would also be convenient to recalculate the indicators?
Every 10 million records take up 1GB of space.

Answer 1

You can use the Logstash + ElasticSearch + Kibana bundle.
Logstash - accepts and parses logs.
ElasticSearch - storage.
Kibana - data visualization.
https :
//www.elastic.cops . I have experience with this kind of connection. Now there are about 1.5 billion events in the repository on 4 servers. Works without problems.

Answer 2

First question: For how long should they be stored? Forever? Is there any sense in this?
The second question: how many visitor records ALREADY and for what period. "Sooner or later" - it can be in a week, and in a year, and never.
If you do not answer these questions in advance, it may turn out that you are solving a problem that does not need to be solved , because the available tools (mysql or even storing logs in CSV) are enough .

Answer 3

Store in MySQL and don't worry

Answer 4

If you really need to solve this problem, then monga + sharding (about volume).

Answer 5

yes in any, you can break it into several tables (for example, by months).