A
A
Alexander2020-09-30 15:58:00
big data
Alexander, 2020-09-30 15:58:00

Architecture for regionally distributed event queue?

Excites a question to the approach of creation of the bus of messages for office + branches. The essence of the problem:
1. There is a certain holding, which includes different legal entities
2. The business of legal entities is different. It can be both production and transport services.
3 I would like to have up-to-date analytics for the entire holding here and now
4. As a way out, merge all data into some kind of read-only database\bus\something and build analytics on this storage.
5. There are requirements - relevance (here and now), availability, reliability

As a solution - just all the data on the wiring, for example, merge somewhere into the bus, from where it all gets into something columnar for aggregation and from where it will be read by something what visualizes in graphs, dashboards and all that.
Let's say it's kafka+vertica+tableau.

But the question arises - where to place the conditional kafka? For example, if there is 1 office and 2 branches, if kafka is located only in the office, the branch may simply not reach you due to a lack of communication. Therefore, I want to make conditionally 2 kafka - in the office and branch, but due to the fact that the branch can be in 2 swoops 3 floods, i.e. small - it will be an overhead. However, it cannot NOT work if the connection suddenly drops. This means that events (messages) need to be accumulated somewhere, so that when the connection is restored, everything will be fine. Who has any experience and recommendations on this topic? (A question about architecture, rather than about the choice of tools)

Answer the question

In order to leave comments, you need to log in

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question