How to conduct distributed computing against a previously unknown algorithm?

Question

Distributed Computing

Good afternoon.
I understand very little about distributed computing. But the task arose to write a software that would allow us to divide calculations between several machines.
At the same time, it is not known in advance what kind of calculations these will be. Anything can be: from parsing large text files to processing a video stream.
Those. often such complex and lengthy operations are performed. And there are several computers idle idle. I would like to use them somehow.
To be honest, I have absolutely no idea how to do this. Please tell me which direction to go? Do you have any ideas or articles on this topic? I would be very grateful for any information.

Answer 1

First, you need to understand whether the work that needs to be distributed can be divided into independent blocks. In some cases, this is quite easy - parsing files can be divided line by line into blocks and parsing each block on your machine. With a video stream it is already more difficult.
If this can be done, then just write a worker and a coordinator. The worker registers with the coordinator, the coordinator breaks the task into small blocks, sends them to the available workers, collects the results back - the task of organizing such interaction is quite trivial in itself.
Those. your main task is to deal with the distributed computing itself, the tasks that can be solved with their help. Here, for example, is an intuit course
www.intuit.ru/studies/courses/1146/238/info