Pool of periodic tasks on the server

Question

linux

Greetings!

There is a script which is executed long enough time. Its task is to collect data, so several instances are launched with different parameters, by cron.

The problem is that it runs at different times, so, for example, if we run 4 instances of the script at 12.00, and the next 4 instances at 18.00, it’s not a fact that those launched at 12.00 have finished working.

The script itself works very well, as it should. The problem is to somehow organize a pool or queue so as not to start new instances while the old ones are running.

Is there some kind of tool so that, say, no more than 4 worker processes work at each moment of time, and new ones are launched in turn with certain parameters? Or do I need to write my own something based on gearman for example? I thought there was a ready tool, but I can't find it.

The makefile -j solution, which is described here stackoverflow.com/questions/463963/parallel-processing-from-a-command-queue-on-linux-bash-python-ruby-whateve , is not clear how to put it on a stream.

Answer 1

unix way - put a file in / var / run / pid and check for its existence. if there is a task not to start
4 different tasks - 4 pid files, you should not mix them if they are clearly separated, and not just 4 different threads of one task.

Answer 2

As I see, everyone invents their own bicycles, instead of using a ready-made solution =)
I, in this case, use flock.
Here is an example:

/usr/bin/flock -x -w 0 /var/lock/my_events/event_name -c /path/to/your/script.sh

And here is a brief description of the parameters:

Usage:
 flock [-sxun][-w #] fd#
 flock [-sxon][-w #] file [-c] command...
 flock [-sxon][-w #] directory [-c] command...

Опции:
 -s  --shared     Get a shared lock
 -x  --exclusive  Get an exclusive lock
 -u  --unlock     Remove a lock
 -n  --nonblock   Fail rather than wait
 -w  --timeout    Wait for a limited amount of time
 -o  --close      Close file descriptor before running command
 -c  --command    Run a single command string through the shell
 -h  --help       Display this text
 -V  --version    Display version

Answer 3

The guys above have more unix-way solutions, but I would use RabbitMQ / 0MQ and so on ...

Answer 4

ps aux | grep script.sh
not?

Answer 5

I always use this wrapper for all scripts to protect against re-launch.
#!/bin/sh
PIDFILE=/var/run/rsync_deploy.pid
if [! -e $PIDFILE ]; then
touch $PIDFILE
rm $PIDFILE
fi

Answer 6

Use lock files.
Something like this:

#!/bin/bash

DEFAULT_LOCK="/tmp/.somelock"

lock() {
  LOCK=${1:-$DEFAULT_LOCK}
  lockfile -r 0 $LOCK 1>/dev/null 2>&1
  return $?
}

unlock() {
  LOCK=${1:-$DEFAULT_LOCK}
  [ -f $LOCK ] && rm -f $LOCK
  return $?
}

After you can write:

lock && echo "lock ok test"
lock || echo "lock not ok test"
unlock && echo "unlock ok test"
unlock || echo "unlock not ok test"

And it will use the standard lock file (handy when you need to run one script).
In your case, you can specify your own lock file, for each script, depending on the parameter(s):

# $PARAM -- что-то уникальное для каждого из 4 запусков скрипта. Например:
# PARAM=`echo -n "[email protected]" | openssl dgst -sha1 -binary | base64`
LOCK="${DEFAULT_LOCK}_${PARAM}"

if ! lock $LOCK ; then
  echo "Locked"
  exit 1
fi

# ваши действия

unlock $LOCK