How to free up RAM on uwsgi webserver?

Question

Nginx

Guys, need some advice. The server on Ubuntu 18 Digitalocean is deployed on flask + nginx + uwsgi. 4 cores 8G RAM. So far, the only operation is the execution of the script for processing the image passed through the form. The operation requires a large amount of RAM during execution. After each iteration of the script, memory is catastrophically reduced. So far, after 5-6 executions, the server crashes with an error due to lack of memory. The RAM consumption log is as follows (the first entry after restarting uwsgi and further after each iteration):

Status of uwsgi processes and memory consumption by processes

I don't quite understand what it's doing. The script processed the picture and gave the result - a number, that's it. The question is, is there a way to free up RAM after the script is executed?
upd
Loading images is standard Flask, it also calls the script for processing detect ()

@app.route('/', methods=['GET', 'POST'])
def index():
    if request.method == 'POST':
        # check if the post request has the file part
        if 'file' not in request.files:
            flash('No file part')
            return redirect(request.url)
        file = request.files['file']
        # if user does not select file, browser also
        # submit a empty part without filename
        if file.filename == '':
            flash('NO selected file')
            return redirect(request.url)
        if not allowed_file(file.filename):
            flash("Only JPG file is used")
            return redirect(request.url)
        if file and allowed_file(file.filename):
            filename = secure_filename(file.filename)
            file.save(os.path.join(app.config['UPLOAD_FOLDER'], filename))
            img = (os.path.join(app.config['UPLOAD_FOLDER'], filename))
            count = detect(images=img)
            return redirect(url_for("result", count=count))
            #return str(count)
    return render_template("index.html")

detect() can be found here https://github.com/ultralytics/yolov3/blob/master/...
pps
Correction, for the purity of the experiment, I launched the image processing script directly from the command line. At the time of processing, it takes about 2G RAM. After the growth operation, there is no memory consumption at all, everything returns to its original value. It turns out that the problem manifests itself after calling the script through the Flask function.

Answer 1

Image processing is best done asynchronously by posting the task to a message queue.
The queue handler will run in a separate process. If image processing takes up a lot of memory, then perhaps it can be heavily optimized.
And what kind of processing, if not a secret? Scaling? Reduction to a single file format?

Answer 2

Nothing concrete can be said without the code. Maybe a memory leak in your code, maybe a leak in the library code, or maybe just pool bloat. The simplest thing is to shift the image processing operation to a separate process.

Answer 3

Perhaps it will be interesting to someone how the struggle with memory ended.
Since the leak was observed when running on a local machine under the native Flask server, the uwsgi settings are discarded. When called directly from the console of the Python detect.py file, everything worked clearly, which means that the processing function has nothing to do with it. Therefore, I decided to follow the advice and run detect.py as a separate thread using Redis Queue (RQ).
Replaced in def index()

count = detect(images=img)
return redirect(url_for("result", count=count))

on the

job = q.enqueue(detect, img)
            while job.result is None:
                pass
            else:
                count = job.result
                return redirect(url_for("result", count=count))

And spun, no leakage. It remains a mystery to me where the memory went, but I am quite satisfied with the result. Plus some new knowledge on Redis, RQ, Supervisor picked up for myself)

Answer 4

Really celery / gearmand and you can process pictures even on a cheap VDS.