S
S
sbh2020-07-12 14:42:31
MySQL
sbh, 2020-07-12 14:42:31

How to implement parallel processing of table records?

There is a table with a huge amount of data.
There is a script that takes an outdated record (the table has a field with the mark of the last processing), processes it and enters the processing result into the table.
There is a lot of data in the table, to speed up processing, I want to run several instances of the script.
But there is a problem - script instances can easily grab 1 and the same record from the table and start processing it at the same time.
How to implement parallel processing of table data and avoid such problems?
So far, the only idea is to add a boolean field "in processing" and change it when one of the script instances needs a record for processing, after processing, change this field back. But here a situation is possible when the script instance dies for some reason and "in processing" remains in the processing state forever.

Answer the question

In order to leave comments, you need to log in

2 answer(s)
S
ScriptKiddo, 2020-07-12
@sbh

Most likely you don't need a new field.
Use SELECT with the FOR UPDATE hint.
If the session dies - MYSQL will automatically roll back the changes within the transaction
https://dev.mysql.com/doc/refman/8.0/en/innodb-loc...

F
Finnick is healthy, 2020-07-14
@phinnik

Make processing in multiple threads, and to ensure that the handlers of each thread do not accidentally take the same thing, use blocking queues

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question