Should I use MongoDB?

Question

ruby

Hello! There is an application that writes real-time logs in decent volumes.
Now writes to a file, for convenient work with them on the web, I want to write them to the database. The logs are of different types, there is no definite structure (the user logged in, entered the command, the server turned off, etc.).
Does it make sense to use MongoDB? Or is MySQL the old fashioned way better?

Answer 1

MongoDB will be more convenient for your task, and the impact on performance will be minimal.

Answer 2

The situation was the same, used MySQL for storage. It worked (in terms of recording, everything is smart). But there was one critical moment. Because there were many message options, and writing a table for each was not the best solution + variable fields. In a word, the structure is described only at the top level. It was necessary to stack something in fields, something in BLOB. As a result, the search did not really work. The handling was heavy. Sometimes dropped out on a timeout.
After it was decided to abandon in favor of MongoDB. Since the transition, several million records have already accumulated. At the same time, it takes only 3.4GB of RAM, on a gig disk (half of these are indexes). Scheduled incremental MapReduce is configured for analysis. The search fits in a second or less. MapReduce (5 aggregations) is completed in a couple of seconds on average.
During the operation, no complaints, a pleasure. In the future, it will be possible to raise the cluster and set up sharding.
So if the structure is not fixed, or rather complex + you need to do various aggregations and data processing, then I recommend looking at MongoDB. As an option, add for a while (two weeks) parallel recording both there and there for analysis.

Answer 3

try memcached)

Answer 4

If the logs are used to generate statistics, then they will have to be translated into SQL one way or another. It all depends on the amount of data. If they are large, you will have to write rotation systems. It is more convenient, of course, to try to first write to clean tables without indexes, and then transfer the data in batches to normal ones, with indexes, for reporting.
IMHO, the data is quite structured. Create a label for each type of logging, or bring it to some kind of denarmalized form that will be normalized during rotation - it's much easier than doing a SQL-NoSql rotation.
If you really are sure that you will never want to see "how many users enter a command that shuts down the server", then you can try mongo.

Answer 5

Use mongo.

Answer 6

I think that it is not necessary to accumulate logs, write to MemCached if the server memory is enough to ensure the recording stream. You can save from MemCached as needed. If the data needs to be accumulated and somehow structured, then MongoDB.
MySQL is completely unsuitable for logging.

Answer 7

I have a bad experience with MongoDB. Due to its nature, the data is stored in JSON format, which affects the size, space ran out wildly quickly, and aggregation and map-reduce over several million records were wildly stupid, sometimes falling out of the timeout, while writing to the database practically stopped working. In short, all the tales about speed turned out to be tales for me, or I absolutely do not know how to cook it.
With MySQL, everything is easier and more reliable. You can take MariaDB - get additional optimizations, and you won't lose anything.

Answer 8

As an alternative, you can consider logstash.net

Answer 9

Not for MySQL in this question, but it has a suitable engine - Archive.
I wonder what is meant by comfortable work. It is also convenient to work with logs in a file.

Answer 10

take hadoop immediately)))

Answer 11

If you already have MySQL in your project and don't have Mongo, it's best to keep using MySQL.
By the way, there are third-party services for managing logs, maybe it's more profitable than raising Mongo instances?

Answer 12

Try Redis