DDwrt1002020-09-28 11:42:24
Apache HTTP Server
DDwrt100, 2020-09-28 11:42:24

Apache Kafka how to make a control channel?

Good day to all. Faced such architectural problem in Kafka.
All the options for using Kafka that I saw on the Internet imply its use as a pipe into which data traffic merges. For example, there is a cluster with logs, we merge the logs, and at the output we have a cluster that threshes these logs. But if you read the product manuals, you can see that the Kafka cluster allows you to use the Point-to-Point ideology. But I did not find examples of how to set up this ideology.

Actually, what is the question. There is a system with many agents, agents collect business data. The collected data is transmitted via Kafka to the processing servers. There are no problems with this. The problem arises with the control channel to the agents. I would like to make it with the help of Kafka. And do not fence another protocol in order to place tasks on agents.

As an option, create your own topic for each agent, then I can guarantee that each agent reads his commands, and cannot get into someone else's command stream, and that all commands will be read. But such a solution from my point of view will not be elegant.
I would like to create something like a topic called Control, where agents are connected, and everyone would definitely receive their message.

Can anyone tell me how to properly implement the agent network control flow from the Kafka side? Tkunt finger on materials to be read, has for such use ?

Answer the question

In order to leave comments, you need to log in

1 answer(s)
Vamp, 2020-09-28

If you want to get by with one topic, then make the number of partitions according to the number of agents and subscribe each agent to your partition.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question