How to avoid wastage of inotifywait and/or implement BASH multi-threaded atomic processing?

Question

bash

The crux of the matter is this. There is a remote server on which files appear in a certain folder in the course of work. There is a remote server that monitors the appearance of these files, in case of occurrence it takes the data, processes it and gives it back. I draw your attention to the fact that the data folder is different than the files being taken. Those. I just pass only the name of the data in this way.

#!/bin/bash
ssh [email protected] inotifywait -e create  /any_path1/ --format "%f" -q -m| while read file; do
    scp [email protected]:/any_path2/$file* ./
  ....
  #некоторая обработка длительностью 5-20 секунд
  ...
  scp $file.res [email protected]:/any_path2/$/result/
done

It should be noted that the processing is very long. And one can only be executed in one thread (i.e., hardware resources are used that cannot be parallelized).
All this works wonderfully if at the same time we have no more than 3-4 files at the same time. Then the files start to get lost.

The question is how to solve the problem of file loss?
One of the solutions that I see is to use multi-threaded launch of the handler. But here I stumble upon another problem: it is necessary to lock each thread until the previous one has completed. And here, no matter how much I searched, I don’t know how to do it in bash and so that it is atomic. Can anyone give some advice? Thank you.

Answer 1

and just mount the folder to the file system no? sshfs