How to get rid of message loss in RabbitMq?

Question

Message Queues

Hello!
We have:

• rabbitmq server
  
• Python script (used by pika) that adds messages to the queue (adding goes in a simple loop of regular id, format 1,2,3,4, ....)
  
• Save-to-disk queue
  

Problem:
When you try to add 100k tasks, not 100k is added, but a little less. I tried to run the cycle several times, each time a certain number of messages are "lost" (from 2k to 4k). Interestingly, if you add sleep to the loop, for example, at 3s after sending every 5k messages, the problem disappears (apparently during this time the rabbit manages to "throw" everything to disk).
Questions:

• Is this behavior of RabbitMq normal (as I understand it, it types messages into memory and does not have time to dump them to disk, which is strange, because there is enough free RAM on the machine and, in theory, all 100k messages can simply be put into memory)?
  
• If this is normal behavior, how to "monitor" it and how to live with it? Maybe you just need to tweak something?
  

Answer 1

- Tune the rabbit config
- Tune the server config
- Monitor logs
- monitor server parameters (disk, percent, memory, simultaneous number of messages)
Stop using the fire and forget principle and control the fact of successful message delivery to the queue with exponential retry in the script

Answer 2

As Ivan mentioned, you need to start controlling the successful receipt of the RabbitMQ message, for which you need to enable a special mode on the send channel. The sending speed in this mode will, of course, be much lower, in order to somehow improve it (provided that the order of messages in the queue is not important), you can:
1) Open several channels and send them in parallel
2) Open a channel without sending confirmation and with confirmation, send, for example, 999 messages to the first one, and then one to the second one and wait until confirmation comes to the last one (something can go wrong, but it looks working).
Additionally, you can try the Lazy mode at the queue.