S
S
Sergey2021-09-03 13:21:16
Messengers
Sergey, 2021-09-03 13:21:16

What technology to use to prevent data loss?

Given:
1. PostgreSQL database;
2. Server for collecting data from equipment via modbus-tcp;
3. Unreliable communication line between 1 and 2;
4. The data collection server continuously transmits data to the database.

Under these conditions, it is necessary to exclude the loss of data between the collection server and the database. That is, if the connection is interrupted, then on the server side the data should be accumulated, and after the connection is restored, all the accumulated data should reach the database. The data source is one and the data receiver is also one.

I'm leaning towards a message broker with a delivery guarantee, but I can't choose, because I don't have any experience with this kind of software. Considered several brokers:
1. Apache Kafka - it turned out that there was no delivery confirmation;
2. RabbitMQ - there is confirmation, but it's a very serious product;
3. Apache MQ - there is confirmation, and it seems simpler at first glance.

Are there any more suitable products?

Answer the question

In order to leave comments, you need to log in

2 answer(s)
D
Drno, 2021-09-03
@Drno

personally, according to my communication with the rabbit - either there is a crooked tech software, or the product itself is so-so ... it falls, does not restart, and stuff like that

S
Sergey, 2021-09-05
@begemot_sun

Let's write a solution in Erlang, shall we?
It will consist of 2 parts - receiver and transmitter.
Can use UDP. As a very complex solution for your task: https://github.com/loguntsov/reliable_udp
I think everything can be done easier.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question