What is the best way to organize a queue of delayed image processing?

Question

PHP

Getting images via php. Then they need to be trimmed / reduced.
I want to make a thresher on a separate server with the possibility of making a pool of such threshers in the future. As well as a pool of servers that receive images and work out the logic. And so that you can pull out / insert any server at any time.
How would you recommend organizing the processing?
Download from the logic server to a random cron thresher? Or send a task to a certain control server and then transfer images to them?
It's a bit intimidating to use and administer several ready-made solutions (such as a radish queue or girman)
I think this is an overhead.
Maybe just slap a server for processing in java? In general, I am grateful for any advice on organization / software

Answer 1

1) RabbtiMQ|Kafka|ActiveMQ|whatever. We put messages in the queue for processing, remember about the restrictions on the size of the message.
2) Self-written solution on crutches and adhesive tape, using a database.
3) Make a decoupling in the form of a data storage service and an interface rest. PHP takes the image, saves it, sends a rest request to process the message stored on the server. LB dynamically distributes requests among threshers.
4) Any variants.
But actually there is a question. Are there many of those images? Maybe you shouldn't bother?

Answer 2

I would make some kind of server-dispatcher with storage and a processing queue (it doesn't matter if it's MQ or something self-written with a database). Its tasks:
1. Interaction with clients - accept input data and queue them for processing (in response, we issue some id), issue the processing status by the passed id (queue, in progress, ready, error) and, when ready, result
2 Interaction with processing servers - transfer received and ready-to-start jobs in turn to server readiness requests for task processing, receive results or error messages, monitor server operation (if some server processes a task for too long or does not send readiness requests, notify the admin, and set the hung task as starting again so that another server intercepts it).
In principle, you can install several dispatchers so that the server-handler interrogates them in turn (for example, the main and the spare).