A
A
Anton2019-03-19 10:46:39
Nginx
Anton, 2019-03-19 10:46:39

How to send nginx logs to ELK in JSON format?

There is a log form nginx:

log_format apatsev escape=json
  '{'
    '"body_bytes_sent":"$body_bytes_sent",'
    '"bytes_sent":"$bytes_sent",'
    '"request_length":"$request_length",'
    '"server_port":"$server_port",'
    '"status":"$status",'
  '}';

server {
    listen 80;
    server_name service-dev.mycompany.io;
    access_log syslog:server=10.233.60.114:5140 apatsev;

With ngrep I see
U 10.2.53.165:38051 -> 10.233.60.114:5140 #18
  <190>Mar 19 10:40:07 dev-int-load-balancer2 nginx: {"body_bytes_sent":"81","bytes_sent":"257","request_length":"520","server_port":"80","status":"200",}

logstash.conf:
input {
  syslog {
        host => "0.0.0.0"
        port => 5140
        type => "log"
        }
  }

filter {
grok {
   match => [ "message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}"]
   overwrite => [ "message" ]
}
geoip {
   source => "clientip"
}

mutate {
   convert => ["response", "integer"]
   convert => ["bytes", "integer"]
   convert => ["responsetime", "float"]
}

date {
   match => [ "timestamp" , "dd/MMM/YYYY:HH:mm:ss Z" ]
}

useragent {
   source => "agent"
    }
}

output {
  elasticsearch {
    hosts => ["127.0.0.1:9200"]
    index => "apatsev-nginx-%{+YYYY.MM.dd}"
  }
}

If you send not JSON, then ELK parses the fields.
If you send JSON, then all JSON is inside the message field.
At least how to remove the line
<190>Mar 19 10:40:07 dev-int-load-balancer2 nginx:
before the JSON ?

Answer the question

In order to leave comments, you need to log in

1 answer(s)
I
ivankomolin, 2019-03-20
@ivankomolin

<190>Mar 19 10:40:07 dev-int-load-balancer2 nginx:
The specified line forms the syslog to which you directed the output of the nginx logs
Dig towards rsyslog templates

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question