M
M
Max2015-12-16 22:40:48
elasticsearch
Max, 2015-12-16 22:40:48

Why does Logstash lose the beginning of a line?

Colleagues, I ask for help.
there is Logstash on AWS.
at the input of files with logs from CloudFront, in the middle of the filter

filter {
        if ([message] =~ /^#/) {
            drop{}
        }
        grok {
        type => "cloudfront"
        pattern => "%{DATE_EU:date}\t%{TIME:time}\t%{WORD:x_edge_location}\t(?:%{NUMBER:sc_bytes}|-)\t%{IPORHOST:c_ip}\t%{WORD:cs_method}\t%{HOSTNAME:cs_host}\t%{NOTSPACE:cs_uri_stem}\t%{NUMBER:sc_status}\t%{GREEDYDATA:cs_referrer}\t%{GREEDYDATA:cs_User_Agent}\t%{GREEDYDATA:cs_uri_query}\t%{GREEDYDATA:cs_cookies}\t%{WORD:x_edge_result_type}\t%{NOTSPACE:x_edge_request_id}\t%{HOSTNAME:x_host_header}\t%{URIPROTO:cs_protocol}\t%{INT:cs_bytes}\t%{GREEDYDATA:time_taken}\t%{GREEDYDATA:x_forwarded_for}\t%{GREEDYDATA:ssl_protocol}\t%{GREEDYDATA:ssl_cipher}\t%{GREEDYDATA:x_edge_response_result_type}"
        }
        mutate {
            type => "cloudfront"
            add_field => [ "_timestamp", "%{date} %{time}" ]
        }
        date {
                type => "cloudfront"
                match => [ "_timestamp", "YY-MM-dd HH:mm:ss" ]
                target => "@timestamp"
        }
}

at the exit, it all falls into elastic. And now, on average, 1 out of 10,000 lines is read crookedly - from the middle. As a result, the fields move out, while outwardly the line looks quite normal.
wtf?

Answer the question

In order to leave comments, you need to log in

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question