Storing user logs?

Question

Journaling

Please share your ideas on how best to keep logs of visits / user actions. It will be faster to write to the database, store in separate files and then write to the database when the server is not loaded.

Answer 1

good old text files + logrotate, no? An interlayer in the form of a database will only create an extra overhead.

Answer 2

it will be faster to store in memcache / hcache or something like that and upload batches to the database / files

Answer 3

It is more reliable to store classic logs - any failures in the statistics processing system will not lead to data loss. However, this applies to systems where statistics are not needed immediately, and where data from an HTTP request is sufficient - data from a session or a database, etc. is not needed.
For example, using nginx, you can fine-tune what and how to log, and such logs can be downloaded, for example, to PostgreSQL with one copy command

Answer 4

If you do not need to “read” the logs often, then text files are the best solution.

Answer 5

It will be faster to write to the database, store in separate files and then write to the database when the server is not loaded.
In my opinion - if there is at least a minimal probability that your service will be of interest to a LARGE number of users at the architecture level, immediately decide that these LOGS are on a separate server (servers) ...

Answer 6

The question is ambiguous. Better test both.
Do you know that if enough large blocks and enough parallel scripts write to a file, then messages can get mixed up?
here there is a need to synchronize access to the file, which means there is a bottleneck where all parallel processes are blocked.
Whereas the DBMS already has some interesting and free mechanisms like INSERT IGNORE that return the result “instantly”.
Prepared statements and a connection pool will also come in handy.
There is also an opinion that a truly loaded project does not need user action logs at all. Except during debugging.

Answer 7

Scribe (http://github.com/facebook/scribe) - quite flexible, fast, simple, aggregation from multiple servers if necessary, possible integration with hadoop