How to use Kafka at all if messages can be duplicated on different receivers?

Question

Java

1. How to build a reliable message processing system if we can get the same message on different subscriber nodes, read it, process it and write the result somewhere. It turns out extra work and it only works with idempodent operations.
   
2. If it somehow manages to distribute messages for processing by only one subscriber (exactly-once option, which is not out of the box), then if I want to filter by key, do I still need to read the entire message?
   

Am I missing something in this system?

Answer 1

It's obvious from your questions that you don't need Publish/Subscribe.
You need Queue. Kafka is the wrong choice as it is a Topic-Based Publish/Subscribe Broker.
Why are Activemq or Rabbitmq not suitable?
1) if your subscribers read from one topic, then duplication cannot be avoided.
2) read from one topic by one subscriber.
Who uses Apache Kafka and how: https://kafka.apache.org/powered-by
https://pbs.twimg.com/media/C3GBWDrXgAAk3Kb.jpg:large

Answer 2

Thank you. It’s just that Kafka is already there, although it’s not clear how to cook it.
I wonder how a pub / sub broker is generally used for data processing, if not for notifications.