How to collect logs?

Question

linux

Need advice on choosing opensors assembly for collecting and processing a large amount of logs. At the moment, 11 windows servers with ASP.NET web applications, 1 server per day gives about 6 gigs of logs in total (2-3 applications). Logs are written to files, files are rotated depending on the application from 100M to 1G per file.
The system should scale easily and transparently with minimal or no downtime, the number of servers will grow, and linux servers will also be added.
We plan to collect logs from files, direct redirection to the collection system is planned directly, but in the indefinite future, therefore it is not relevant at the moment.
Log alerts are not particularly needed, we will not try to replace the system with this assembly. The main requirement is the reduction of logs by timing between machines and applications and the subsequent selection by keys.
Need advice on assembly, with justification of the benefits. For example, ELK or graylog2, which is better and why, what accelerators can be used, like rabbitrq?
You also need advice on choosing an axis and the optimal conf from which you can start testing.

Answer 1

as a person who has encountered ELK
6G per day - not terrible for ELK. But get ready for gluttony from memory. Vsezh Java...
ease of scaling - basically yes. Adding a node to the cluster is a matter of minutes. Synchronization, however, will take time. But this is the case everywhere, the data "by magic" from node to node will not fly.
collect from files - also without any problems. there are both Beats and Logstash - both ideologically correct, from the elastic itself. Yes, there are many alternatives. Up to the python script - pushing into the elastic is not a difficult thing.
mixing and searching - in full growth. fast disks (or better SSD) + plenty of memory and everything will fly.
accelerators - with your 4-5 mb per minute, accelerators are unlikely to be needed.
but what you should think about in advance is what you are going to do with what fields. Otherwise, they will save the file size as a string - and then worry that the search for less does not work anymore. And it's worth thinking about analyzing / not analyzing. Search in an unparsed field is noticeably less gluttonous - and therefore faster

Answer 2

You can't do without a message broker. With growing volumes, sooner or later you will "fall on your nose".
https://kafka.apache.org/ suits your needs like nothing better. Both in volume and performance.
The main scenario is described here:
https://www.elastic.co/blog/just-enough-kafka-for-...
If you want to create something like a log archive. In this scheme, this is just another consumer.