N
N
nurzhannogerbek2018-10-27 07:38:18
elasticsearch
nurzhannogerbek, 2018-10-27 07:38:18

How to update data in index using Logstash?

Hello comrades! Please help me figure it out.
There is a table in a PostgreSQL 10 database . Plus or minus seven thousand records pour in every hour.
Based on this table, I created an index in Elasticsearch using a Logstash script . It looks like this:

input {
    jdbc {
        jdbc_connection_string => "jdbc:postgresql://@host:@port/@database"
        jdbc_user => "@username"
        jdbc_password => "@password"
        jdbc_driver_library => "C:\postgresql-42.2.5.jar"
        jdbc_driver_class => "org.postgresql.Driver"
        statement => "SELECT * from table_name"
    }
}
output {
    elasticsearch {
        hosts => ["localhost:9200"]
        index => "table_name"
    }
}

Questions :
  1. How to update an existing index with new data that comes into the table?
  2. What is the maximum amount of data an index can store? Is it possible to overflow?

Answer the question

In order to leave comments, you need to log in

1 answer(s)
C
chupasaurus, 2018-10-27
@chupasaurus

  1. Either collect the Exported_to_ES column in the table, or let the application start pouring data into Elasticsearch as well.
  2. 2^32 documents per index shard.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question