X
X
Xander1872015-06-19 11:07:33
Perl
Xander187, 2015-06-19 11:07:33

How to scatter task queue across hosts and threads in Perl?

The bottom line is this: there is a pack of commands that need to be launched in a queue with their spreading over hosts into several threads on each (For example, there are 3 hosts, on each you need to run three commands at a time). Now I've implemented it through a bicycle on OpenSSH::Parallel: we connect to the necessary hosts, and already on each of them, through xargs, we start up the whole pack of command lines. Each command has a crutch check for execution (if a folder with the name of the command has already been created, it means that it is running on another host or has already been executed, and we skip it). The trouble is that, firstly, the implementation is very bad, and secondly, sometimes the same command runner manages to run on 2 servers at once.
Actually, how to scatter commands among hosts from a common queue, so that several pieces are executed on each server?

Answer the question

In order to leave comments, you need to log in

2 answer(s)
E
Egor Kazantsev, 2015-06-19
@saintbyte

And what thread gearmand does not roll? I launched as many workers as needed, and the workers are already running commands.

M
microphone, 2015-06-19
@microphone

I'm certainly not very into parallel, but what about the task scheduler?

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question