Why isn't filebeat sending nginx logs?

Question

Nginx

filebeat configured. without the connected nginx module, the logs come to elasticsearch. The logs also came when the pipeline of the nginx module was used by default. Got errors

Provided Grok expressions do not match field value:

. In Grok Debugger, I found out and fixed a parsing error, after making changes to the pipeline and updating it in ingest, the logs either stopped coming or were no longer displayed in kibana.
filebeat started like this:

./filebeat -c filebeat.yml --modules=nginx

in the logs

2019-03-14T18:19:31.236+0300 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu": {"system":{"ticks":710,"time":713},"total":{"ticks":2640,"time":2651,"value":2640},"user":{"ticks ":1930,"time":1938}},"info":{"ephemeral_id":"3a00ee7c-f02e-4c30-8efc-9ea00ab9fe73","uptime":{"ms":840022}},"memstats": {"gc_next":4249088,"memory_alloc":2199672,"memory_total":86654152,"rss":16384}},"filebeat":{"events":{"added":30,"done":30}, "harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":1}},"output":{"events":{"acked":30,"batches":10,"total":30} ,"read":{"bytes":3633},"write":{"bytes":22440}},"pipeline":{"clients":5,"events":{"active":0,"published ":30,"total":30},"queue":{"acked":30}}},"registrar":{"states":{"current":2,"update":30},"writes ":10},"system":{"load":{"1":0.07,"15":0.17,"5":0.11,"norm":{"1":0.0022,"15":0.0053, "5":0.0034}}}}}}acked":30,"batches":10,"total":30},"read":{"bytes":3633},"write":{"bytes":22440}},"pipeline":{"clients ":5,"events":{"active":0,"published":30,"total":30},"queue":{"acked":30}}},"registrar":{"states" :{"current":2,"update":30},"writes":10},"system":{"load":{"1":0.07,"15":0.17,"5":0.11, "norm":{"1":0.0022,"15":0.0053,"5":0.0034}}}}}}acked":30,"batches":10,"total":30},"read":{"bytes":3633},"write":{"bytes":22440}},"pipeline":{"clients ":5,"events":{"active":0,"published":30,"total":30},"queue":{"acked":30}}},"registrar":{"states" :{"current":2,"update":30},"writes":10},"system":{"load":{"1":0.07,"15":0.17,"5":0.11, "norm":{"1":0.0022,"15":0.0053,"5":0.0034}}}}}}published":30,"total":30},"queue":{"acked":30}}},"registrar":{"states":{"current":2,"update":30}," writes":10},"system":{"load":{"1":0.07,"15":0.17,"5":0.11,"norm":{"1":0.0022,"15":0.0053 ,"5":0.0034}}}}}}published":30,"total":30},"queue":{"acked":30}}},"registrar":{"states":{"current":2,"update":30}," writes":10},"system":{"load":{"1":0.07,"15":0.17,"5":0.11,"norm":{"1":0.0022,"15":0.0053 ,"5":0.0034}}}}}}

Tried to run like this:

./filebeat -c filebeat.yml

My settings are empty in the filebeat logs :

filebeat.yml

filebeat.prospectors:
- input_type: log
paths:
- /somelog/access.log
logging.level: debug
output.elasticsearch:
hosts: ["host:9200"]
setup.kibana:
host: "host:5601"
username: "kibana "
password: "kibanamama"
filebeat.config.modules:
enabled: true
path: ${path.config}/modules.d/*.yml

paths:
-
/somelog/access.log the correct ones are specified in the nginx module settings

../modules.d/nginx.yml

- module: nginx
# Access logs
access:
enabled: true
var.paths:
- /opt/ibm/appuser/nginx/nginx-1.14.2/logs/access.log
# Error logs
error:
enabled: true
var.paths:
- /opt/ibm/appuser/nginx/nginx-1.14.2/logs/error.log

default.json

{
"description": "Pipeline for parsing Nginx access logs. Requires the geoip and user_agent plugins.",
"processors": [{
"grok": {
"field": "message",
"patterns":[
"\"? %{IP:nginx.access.remote_ip} - \\[%{TIMESTAMP_ISO8601:nginx.access.time}\\] \"%{WORD:nginx.access.method} %{DATA:nginx.access.url} HTTP /%{NUMBER:nginx.access.http_version}\" %{NUMBER:nginx.access.response_code} %{NUMBER:nginx.access.body_sent.bytes} \"%{DATA:nginx.access.referrer}\" \ "%{DATA:nginx.access.agent}\" \"%{DATA:nginx.access.remote_ip_list}\""
],
"pattern_definitions": {
"IP_LIST": "%{IP}(\"?,?\\s*%{IP})*"
},
"ignore_missing": true
}
}, {
"split": {
"field": "nginx.access.remote_ip_list",
"separator": "\"?,?\\s+"
}
}, {
"script": {
"lang": "painless" ,
access.remote_ip = ctx.nginx.access.remote_ip_list[0]; }"
}
}, {
"remove":{
"field": "message"
}
}, {
"rename": {
"field": "@timestamp",
"target_field": "read_timestamp"
}
}, {
"date": {
"field": "nginx.access.time",
"target_field": "@timestamp",
"formats": ["dd/MMM/YYYY:HH:mm:ss Z"]
}
}, {
"remove": {
"field": "nginx.access.time"
}
}, {
"user_agent": {
"field": "nginx.access.agent",
"target_field": "nginx.access.user_agent"
}
}, {
"remove": {
"field": "nginx.access.agent"
}
}, {
"geoip": {
"field": "nginx.access.remote_ip",
"target_field": "nginx.access.geoip"
}
}],
"on_failure" : [{
"set" : {
"field" : "error.message",
"value" : "{{ _ingest.on_failure_message }}"
}
}]
}

Farmat logs access.log

10.127.238.137 - [2019-03-14T17:54:23+03:00] \"POST /nba/checkNewCalls HTTP/1.1\" 200 30 \" https://somhost:8453/test/index?tab=currentCommuni ... " \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36\" \"-\"