What is the best database to write a lot of site logs to?

Question

Database

I had a task to write my own system for collecting site visit statistics, something like Yandex Metrics and Google Analytics.
There will be many sites in the pool, I think the total attendance is up to 100k users per day.
The scheme will be similar: a js script is issued, which is hung on the site, it collects the necessary data and sends a record to the server. The tables will contain approximately 50 visitor parameters (IP, User Agent, browser parameters...)
The question arose: how is it more expedient to store data? Which database to choose? Mysql, Postgresql, Mongodb, Redis or friend?
The system statistics will give out not only the value of the visit itself, but it can also be grouped according to different parameters (like in Yandex Metrica).
I also understand correctly that it is enough to make entries in the database from a js script with ordinary ajax requests, or has a more efficient scheme been invented so that the server does not fall by 100k records per day?

Answer 1

clickhouse , which was specially designed for Yandex.metrics

Answer 2

ELK

Answer 3

Just as correctly, I understand that it is enough to make entries in the database from a js script with ordinary ajax requests

No. Must be a web server. And some layer that can pull data from the database and give the result to the web server.

Answer 4

Well, in addition to clickhouse, I also recommend kafka with an entry in the same slickhouse or hbase / hadoop. Moreover, data can be pulled out of kafka at any time and for any time (well, you can set it up as a retention). So write in kafka, and aggregate wherever God sends, even to hbase, even to clickhouse, even to postgres.

Answer 5

I read that PostgreSQL now knows how to store data in columns, like clickhouse.

Answer 6

In order not to put the server, it is better to write to a file, then once every half an hour or ten minutes (for example) export to a relational database (postgresql or mariadb is not much difference for your task).
Also provide archival tables (for example, by months or years).
This is how piwik did it, or now it's called matomo.
By the way, why don't you want to use piwik?

Answer 7

you can also look at ready-made solutions:
https://medevel.com/best-20-open-source-free-self-...
For example, we use matomo / demo