Which is better to use for parser logs: mysql or elasticsearch?

Question

MySQL

I am making a parser and the question arose of how to save the previous parsing results. at the moment there is one table in the database where the results of the last parsing are stored. those. before starting the parser, all data is removed from this table and new ones are added. It is necessary to save somewhere completely all the data of previous parsers, i.e. keep logging, so that later you can do some statistics with them (for example, how the price changed). how best to do it? there will be quite a lot of data from the work of parsers, so I thought to store them in elastic, since it will be able to process large amounts of data faster. Or is it somehow possible to solve at the mysql level so that there are no problems in the future? prompt with advice

Answer 1

Elastic looks like an overkill here, it's quite possible to get by with just mysql, just not deleting the data / writing it to the table. Well, "quite a lot" is a very different concept can be.
The processing speed of elastic and mysql is not entirely correct to compare, elastic is more likely for (in) a clear search by data - if there is a clear understanding of which field you will search for (for example, by product ID) - then mysql is no worse and much easier to set up and operate

Answer 2

I would recommend elastic. Firstly, it is much more convenient to store logs in it, you can set up indexes and make them in a cluster, if the log format changes, then it’s enough just to make several different data schemes. Search and aggregation are much more powerful than in SQL. You can make aliases and split indexes as you like.
Secondly, you can attach kibana, or grafana, or even write something of your own to the elastic - you can even pull the REST API through curl.
Well, if elastic is not enough, then druid.io will help you or kafka :-)

Answer 3

What to use?
Depends on the load, one of the advantages of ElasticSearch is the compression of stored data, not a clear search, horizontal scaling out of the box (without dancing with a tambourine), fault tolerance when the cluster is correctly configured
.