How to parse log using Logstash?

Question

linux

There is a log:

{«message»:»1353480: CORE-DSS: Mar 10 06:22:06.711: %SEC-6-IPACCESSLOGP: list inside_DSS denied udp 10.22.110.61(60980) -> 255.255.255.255(1947), 1 packet «,»@version»:»1″,»@timestamp»:»2015-03-10T07:11:31.697Z»,»type»:»all_log»,»host»:»10.23.0.200″}

The output should be at least:

denied udp 10.22.110.61(60980) -> 255.255.255.255(1947)

Ideally like this:

2015-03-10T07:11:31 denied udp 10.22.110.61(60980) -> 255.255.255.255(1947) host 10.23.0.200

I write a simple config:

input {
udp {
type => «all_log»
port => 3000
}
}
filter {
grok {
type => «all_log»
pattern => «%{WORD} %{PROG} %{NOTSPACE} -> %{NOTSPACE}»
}
}
output {
elasticsearch {
embedded => «true»
host => «localhost»
}
file {
path => «/srv/LOG/LOG_all/all.oblast.log»
}
}

And it doesn't work - Kibana hangs! What's wrong? And how to make the second option? Tell me please! The solution to this problem will be a good example for me! And also, what knowledge needs to be improved in order to write filters. There is a book on logstash, but it's not free. Thanks in advance for your reply!

Answer 1

I would advise you to break the message into parts completely through the filter, like hello, this is how they do it for IIS logs

grok {
    # check that fields match your IIS log settings
    match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} %{IPORHOST:site} %{WORD:method} %{URIPATH:page} %{NOTSPACE:querystring} %{NUMBER:port} %{NOTSPACE:username} %{IPORHOST:clienthost} %{NOTSPACE:useragent} %{NUMBER:response} %{NUMBER:subresponse} %{NUMBER:scstatus} %{NUMBER:time_taken}"]
  }

You send the entire message to the database, but make the fields you need, and then you will use them to make a filter and dashboards. Grok wrote it himself - I would advise you to use the debugger https://grokdebug.herokuapp.com/
Also, do NOT immediately look into kibana, but deal with elasticsearch. To get started, let's go watch the video https://www.youtube.com/watch?v=7FLXjgB0PQI and also install Sense in chrome and learn how to write queries.
I also see that you send all messages not only to the database, but also to the file:

file {
path => «/srv/LOG/LOG_all/all.oblast.log»
}

Is it empty or does it have messages?

Answer 2

Online debugger for patterns:
https://grokdebug.herokuapp.com/

Answer 3

The example message is taken from a file. Messages to the file go only when the lines related to filter are commented out (ie, when the filter is disabled). I watched the video but didn't understand what it was about. Your idea about breaking the message into parts completely, I caught it. For my filter, I just used the site you specified and I can break the entire log message into parts. But again, the question is what to do with this partition, how to get the desired result?
Installed Sense for Crome in the line SERVER localhost:9200 I write instead of localhost the IP address of the server and specify port 9200, but nothing happens. The string is returned to localhost:9200.