M
M
Mamol272021-09-06 16:53:37
JSON
Mamol27, 2021-09-06 16:53:37

How to parse json in logstash?

Hello.
I am fetching data from a table from a database using the JDBC Source Connector in a Kafka topic and fetching it with logstash.
data is obtained in the following form:

spoiler
"{\"schema\":{\"type\":\"struct\",\"fields\":[{\"type\":\"int64\",\"optional\":false,\"field\":\"FIELD_1\"},{\"type\":\"int64\",\"optional\":false,\"field\":\"FIELD_2\"},{\"type\":\"int32\",\"optional\":true,\"field\":\"FIELD_3\"}],\"optional\":false,\"name\":\"TABLE\"},\"payload\":{\"FIELD_1\":20,\"FIELD_2\":4,\"FIELD_3\":52}}"


With json filter in logstash I get

only json
{
    "@timestamp" => 2021-09-06T09:34:45.914Z,
       "message" => {
        "payload" => {
            "FIELD_2" => 2,
            "FIELD_1" => 26,
            "FIELD_3" => 57
        },
         "schema" => {
              "fields" => [
                [0] {
                       "field" => "FIELD_1",
                    "optional" => false,
                        "type" => "int64"
                },
                [1] {
                       "field" => "FIELD_2",
                    "optional" => false,
                        "type" => "int64"
                },
                [2] {
                       "field" => "FIELD_3",
                    "optional" => true,
                        "type" => "int32"
                }
            ],
                "name" => "TABLE",
            "optional" => false,
                "type" => "struct"
        }
    },
      "@version" => "1"
}

That is, the values ​​​​of the fields that interest me are numeric.

After filter
filter
filter {
    json {
        source => "message"
        target => "message"
    }
    mutate {
        add_field => {
            "FIELD_1"  => "%{[message][payload][FIELD_1]}"
            "FIELD_2"  => "%{[message][payload][FIELD_2]}"
            "FIELD_3"  => "%{[message][payload][FIELD_3]}"
        }
        convert => {
            "FIELD_1"  => "integer"
            "FIELD_2"  => "integer"
            "FIELD_3"  => "integer"
        }

        remove_field => [ "message" ]

    }
}


output
{
        "FIELD_3" => "52",
        "FIELD_2" => "4",
    "@timestamp" => 2021-09-06T13:01:58.448Z,
        "FIELD_1" => "20",
      "@version" => "1"
}

However, I am trying to get the view structure
needed
{
    "@version" => "1",
    "FIELD_1" => 20,
    "FIELD_2" => 4,
    "FIELD_3" => 52
    "@timestamp" => 2021-09-06T09:34:45.914Z
}

For some reason, the fields remain text, I don’t understand how to take the values ​​​​not through sprintf.
Is there a more elegant way to parse this data into the form I need?

Answer the question

In order to leave comments, you need to log in

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question