S
S
SlavikF2019-11-01 08:27:27
Nginx
SlavikF, 2019-11-01 08:27:27

How to configure Filebeat to send data to a specific pipeline?

There is a web server (Ubuntu) running Nginx + PHP.
This server has Filebeat, which sends web server logs to the Elastic Ingestion node. That is, without Logstash, directly from Filebeat to Elastic. There isn't much traffic, so it's ok.
When I installed Filebeat, I tweaked the default pipeline a bit so that Russian letters in the URL are displayed correctly (urldecode).
Works all several months, stably. But every time I do an "apt upgrade" and if filebeat is updated, it starts writing to a new index, in which a default pipeline is created without my edits.
Q: How do I configure the Filebeat Nginx module to use my custom pipeline?
Now when I look at all my pipelines, here is how many there are:
elk.slavikf.com:9200/_ingest/pipeline/filebeat-*

filebeat-nginx-access-default: {},
filebeat-7.3.1-nginx-error-pipeline: {},
filebeat-7.4.1-nginx-error-pipeline: {},
filebeat-7.2.0-nginx-access-default: {},
filebeat-7.3.2-nginx-error-pipeline: {},
filebeat-7.4.1-nginx-access-default: {},
filebeat-7.3.1-nginx-access-default: {},
filebeat-7.3.2-nginx-access-default: {},
filebeat-7.2.0-nginx-error-pipeline: {}

filebeat-nginx-access-default - I created this, my custom pipeline, and the rest, as I understand it, are created automatically.
I tried to write this in my nginx.yml, but it doesn't help:
- module: nginx
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/nginx/*/*access.log"]

    # Convert the timestamp to UTC
    var.convert_timezone: true

    # The Ingest Node pipeline ID associated with this input. If this is set, it
    # overwrites the pipeline option from the Elasticsearch output.
    output.elasticsearch.pipeline: 'filebeat-nginx-access-default'
    pipeline: 'filebeat-nginx-access-default

Created a bug with Elastic: https://github.com/elastic/beats/issues/14348 can't specify pipeline for module
But so far they are silent about something.

Answer the question

In order to leave comments, you need to log in

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question