D
D
dimonchu2019-10-27 19:45:44
Message Queues
dimonchu, 2019-10-27 19:45:44

Queues, or get a job and delete it right away?

For example, there are two scripts that work asynchronously and take 50 objects from the database at the same time.
The question is how to make sure that the scripts do not take the same objects and do not process the same thing each?
script 1 -> db -> 50 first objects
script 2 -> db -> 50 first objects
I originally thought about queues.
script 1 -> Queue -> First object
script 2 -> Queue -> Next object
But I don't understand how to make the queue remove the job from it as soon as the script has taken this object.
You still have to: take the object -> process -> delete. But at this moment, another script can take the same job, while the first script will process it.
I want by analogy with the php array_pop function:
What am I doing wrong?)

Answer the question

In order to leave comments, you need to log in

1 answer(s)
R
Roman Mirilaczvili, 2019-10-30
@2ord

One dispatcher process periodically checks for pending records and queues a 50-record array of primary keys.
Those. the process makes a selection of 50 unprocessed records, puts it in the queue, and so on until there are less than 50 left. It wakes up after a while and repeats this process again.
Records that are being processed must be type-tagged dispatched- this ensures that the records will not be selected again.
The workers each take their task with pk arrays, process them and mark the records with a status done, and so on until they run out.
And you can do without the dispatcher. The main thing is to mark the status of the record.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question