K
K
Konstantin Vlasov2021-12-26 17:11:26
Docker
Konstantin Vlasov, 2021-12-26 17:11:26

How to fix LEADER_NOT_AVAILABLE error in Console consumer and producer Apache Kafka?

I'm trying to deploy Kafka to Docker using docker-compose.
I'm using a Macbook Pro M1 with Ubuntu 20.04 VM.
I took this guide and the corresponding source code as a basis (disabled authorization and changed images, since the images in the article are for x86 machines).
Zookeeper runs in a separate container, Kafka in a separate one.
Everything starts, but when you try to start the Console consumer with the following script

/home/ubuntu/kafka/bin/kafka-console-consumer.sh --topic new_topic --from-beginning --bootstrap-server 192.168.0.106:9092


many times an error is displayed :

[2021-12-26 16:36:00,362] WARN [Consumer clientId=consumer-console-consumer-32734-1, groupId=console-consumer-32734] Error while fetching metadata with correlation id 3 : {new_topic=LEADER_NOT_AVAILABLE} ( org.apache.kafka.clients.NetworkClient)

when trying to create a producer using the following script:

/home/ubuntu/kafka/bin/kafka-console-producer.sh --topic new_topic --bootstrap-server localhost:9092

producer is created, however , when trying to send a message many times, an error is displayed in the console:

[2021-12-26 16:09:53,385] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 5 : {new_topic=LEADER_NOT_AVAILABLE} (org. apache.kafka.clients.NetworkClient) I

use the following code in the docker-compose.yml file (all images from docker hub):

version: '2'
services:
  zookeeper:
    image: zookeeper
    expose:
      - "2181"
  kafka:
    image: fogsyio/kafka:2.2.0
    ports:
      - "9092:9092"
    expose:
      - "9093"
    environment:
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: 'true'
#      KAFKA_ADVERTISED_PORT: 9093
#      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka:9093,OUTSIDE://192.168.0.106:9092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_LISTENERS: INSIDE://0.0.0.0:9093,OUTSIDE://0.0.0.0:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE
      KAFKA_SASL_ENABLED_MECHANISMS: PLAIN
      KAFKA_SASL_MECHANISM_INTER_BROKER_PROTOCOL: PLAIN
    depends_on:
      - zookeeper


commented out the lines that I tried to add to correct the situation, they did not help. I came across posts where it is said that this is due to the fact that at the time Kafka was launched, the topic does not exist and when the message is resent, it will be sent, but this does not happen even after the producer / consumer is restarted.

What else can you try? I would like to ask the author of the original post a question in the comments, but my account only allows commenting on fresh posts(

Answer the question

In order to leave comments, you need to log in

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question