L
L
luxter2017-03-13 18:04:34
linux
luxter, 2017-03-13 18:04:34

Setting up an ELK bundle for central logging, how to accept logs from rsyslog clients?

Good day! There is a task to lift a sheaf on collection of dens through UDP from clients. I'm doing a trial test version, but pitfalls are everywhere.
Available:
stream of logs from several clients (*.*@logstash-server);
logstash configured to receive and send to elasticsearch + kibana.
logstash config (essentially copied from official documentation for receiving and processing logs from rsyslog clients)

input {
    udp {
      type => rsyslog
      port => 50514
          }
}

filter {
 if [type] == "rsyslog" {
     grok {
     match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %
     add_field => [ "received_at", "%{@timestamp}" ]
     add_field => [ "received_from", "%{host}" ]
     }
     date {
     match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
     }
                                           }
}

output {
     elasticsearch { hosts => ["<hostname><port>]
     index => "logs-%{+YYYY.MM.dd}"
     }
}

Problem: the logs are pouring in if you look at the output of tcpdump. Ports are open. Logstash is listening on the correct port.
But the logs are nowhere to be seen. They rain down on the interface, but where are they saved? And are they saved at all?
I did not make any settings in elasticsearch, except for getting access through the browser.
How to correctly accept logs from rsyslog clients?
I did not find a single manual where logstash would be used as a receiver from rsyslog and then sent to elasticsearch.
I would appreciate any tips and/or hints.

Answer the question

In order to leave comments, you need to log in

1 answer(s)
L
luxter, 2017-03-15
@luxter

I found the problem, it turned out to be simple. Simplified basic config:

input { udp {
    port => 50514
        type => syslog} }

filter {
  if [type] == "syslog" {
      grok {
           match => { "message" => "%{SYSLOGBASE} : %{GREEDYDATA:syslog_message}" }
           }
      syslog_pri { }
                         }
        }

output {
  elasticsearch { hosts => ["ИМЯ ХОСТА:9200"] }
    stdout { codec => rubydebug }
    }

PS Logstash listens on 50514 and iptables redirects incoming traffic to port 514:
iptables -t nat -I PREROUTING -p udp --dport 50514 -j REDIRECT --to-port 514

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question