RabbitMQ: how to properly perform heavy tasks?

Question

PHP

Hello!
At our project, jobs are sent to generate reports in the rabbitmq queue.
The queue itself is served by a php-consumer (works via php-amqplib).

The problem is that generating reports can take 10 minutes, some even an hour. The

consumer works as follows:
it takes N messages from the queue and starts generating reports immediately.

At the moment when the consumer is busy generating a report, as I understand it, rabbitmq sends some checks for the viability of the consumer - heartbeat . By default, if the consumer does not respond for 60 seconds, then at the next iteration an error occurs that the connection is lost.

I have two questions:

1. What is the risk of disabling - heartbeat in rabbitmq?I read it here but still don't understand.
2. How is it customary to handle the heavy tasks that the costumer receives? maybe they should not be launched immediately in the consumer itself? if you run it as a separate process, then the question immediately arises, what to do if an error occurs in the process of generating a report? the message from the queue is read and sent to ack.

best practices needed))

Answer 1

The consumer itself does not have to process tasks. Its main purpose is to unload the queue and send tasks further, for example, to a database or to other servers through a balancer. The first way is best.
Next, a worker looks at the unprocessed tasks in the database and slowly processes them. Processed tasks are flagged.

Answer 2

There was a similar problem, the sishny client in one flow was not able normally in heartbeat. When disabling heartbeats at the AMQP protocol level, there were problems with "hanging clients" on the RabbitMQ side, when the application exited unexpectedly. Setting TCP keepalive helped to solve this problem.
I don’t know exactly how modern php works, but if it is possible to keep a separate thread to maintain an AMQP connection with heartbeat enabled, then it will be better than via TCP keepalive. It's better to send ack back after the task is finished processing.

Answer 3

This looks like a problem with the php-amqplib library, php still doesn't know how to multithread?
You can create a process that will generate a report.

Answer 4

And how do you maintain a connection to the database if there are no requests to it during the report generation? We do a ping to the database, as well as to the rebitt, thereby reporting that the cosumer is alive and working.

Answer 5

Partial solution https://github.com/php-enqueue/enqueue-dev/blob/ma...

Answer 6

php-amqplib nerdy, better amqp extension from pecl