D
D
Dmitry2017-11-21 11:10:23
RSYSLOG
Dmitry, 2017-11-21 11:10:23

How to organize the collection of logs on the highload?

We tried this scheme:
app->rsyslog->filebeat---[tls network]--->logstash->graylog->elasticsearch
Load up to about 50 million messages per day. Logstash is dead.
I think to make it so that kibana and graylog would not write anything to the elastic, ideally, but only visualize the data.
Does it make sense to start logstash locally on hosts and write to it directly from rsyslog via json template. And would logstash'y put everything in elastic? (similar to this approach )
I would like to hear the opinions of professionals, how do you manage logging?

Answer the question

In order to leave comments, you need to log in

2 answer(s)
K
kot-airplane, 2017-11-21
@kot-airplane

Maybe it will help: Yury Nasretdinov, "Collecting logs in the "cloud" in Badoo"

D
Dimonchik, 2017-11-21
@dimonchik2013

experiment,
aggregate
Clickhouse to help, Sphinxsearch to help (however, after the clickhouse with its %find% is no longer strong), GoogleBigData to help
, of course, there is a loss in the network: consolation - sampling to help, or selective logging

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question