What are the efficient ways to transfer data (2 MB) from one application to another in linux?

Question

linux

There is a web-app that receives a file from the user in a Post request:
1. Saves the file (1 MB) to fs along a unique path: uniquePath := userID + time
2. Passes the path to the file to the z-app program: exec.Command(Z , uniquePath).Stdout = &stdout
3. Waits for the z-app program to complete (about 5-10 seconds, web-api requests)
4. Sends the result (2 MB) to Post: w.Write(stdout.Bytes())
Program z-app:
1. Will be updated frequently, with no loss to waiting users.
How to effectively reorganize the interaction of web-app with z-app? Pipe, FIFO, something else?
web-app and z-app are written in Golang

Answer 1

If your z-app is essentially a backend for a web app, I don't see a problem running z-app as a subprocess, feeding it data via stdin and taking the result from its stdout, Since z-app takes a long time to run anyway, the overhead of starting a child process will not be a problem. Python does this with its subprocess module, like golang does - I don't know.
In any case, you will end up with transferring data either directly through the stream, or saving the stream to disk in a file and passing the file name. If your z-app is frequently updated, the solution with subprocesses will also give you no problems with updates - a new instance will simply start from a new file

Answer 2

What is the problem with outputting an error to the console? If errors are output to STDERR, and data is exchanged via STDOUT, there will be no problems.
It is more convenient to organize communication via Unix sockets or via tcp to localhost. Let z-app hang as a daemon and accept tasks for processing.

Answer 3

If the latency of 5-10 seconds suits you, you can get by with fsnotify (for example , https://fsnotify.org/). web-app spawns files along a unique path. z-app keeps track of file system changes and processes new ones. The result of the work of z-app is uploaded to the same or a new file along a unique path. web-app, in turn, monitors the file system and picks up new results. Everything is asynchronous, there is no direct connection at all. z-app can run in multiple threads.

Answer 4

linux? Well, start with this
D-Bus
CORBA
Well, obviously.