G
G
GrossmasterR2020-01-29 08:25:01
Java
GrossmasterR, 2020-01-29 08:25:01

How to monitor changes in databases?

Dear habrchane, hello.
Need your advice.

Algorithm for determining data schema changes.
Introduction:
In a large system that implements a minroservice architecture, it often becomes necessary to synchronize hot data between various components (executable modules). As a rule, a single point of truth for all components is the DBMS, namely, some schema or a set of DBMS objects. But frequent calls to these objects, especially queries containing aggregate or analytical functions, negatively affect the performance of the DBMS as a whole. Therefore, for different situations, various tactics are developed to determine the difference in the states of certain DBMS objects for synchronization with the application.
What should be done:
Design an algorithm (function) that determines whether there are changes in the DBMS or not starting from some time or identifier.
Variations:

  • Calculation change the entire circuit.
  • Calculation of changes in given objects (tables, columns).
  • Calculation implemented only in JAVA without using DBMS system objects.
  • (Advanced) Calculation implemented only in SQL on Postgres DBMS.
  • The calculation allows the introduction of additional entities in the data schema, or recommendations for schema design.

Evaluation criteria:
The code must be covered by load tests, evaluations of query plans are given.

It is written relatively complicated, the essence, as I understand it, is to write some simple implementation in java: a class that will track data changes.

I googled and googled and this and that. Someone suggests, based on the analysis of the transaction log, to use the mechanism of triggers. It seems that MS Sql Server and postgresql have some built-in tools for this, someone suggested doing this using checksums, at the same time I could not find something simple available. Maybe there is a hint?

Answer the question

In order to leave comments, you need to log in

1 answer(s)
S
Sergey Gornostaev, 2020-01-29
@GrossmasterR

The question has already been asked: Is there a way to get the state update of a postgresql table by an external java process?
But it is better to use a queue broker to propagate changes, a DBMS is not the best tool for this.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question