How to make Apache Kafka work with large files?

Question

JSON

Hello! I'm trying to upload a file larger than 1MB to Kafka and I get the following error:

ERROR Error when sending message to topic test_topic with key: null, value: 6846219 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache .kafka.common.errors.RecordTooLargeException: The message is 6846307 bytes when serialized which is larger than 1048576, which is the value of the max.request.size configuration.

Used KAFKA:
https://hub.docker.com/r/bitnami/kafka/
https://hub.docker.com/r/wurstmeister/kafka/

Answer 1

https://hub.docker.com/r/wurstmeister/kafka/
" if you want to customize any Kafka parameters, simply add them as environment variables in docker-compose.yml, eg in order to increase the message.max.bytes parameter set the environment to KAFKA_MESSAGE_MAX_BYTES: 2000000."