/bin/bash: Too many open files in system. If you do not reboot the server, will it pass after some time? How to avoid?

Question

linux

I understand my mistakes, but I can not find a method to avoid the server hanging.
I ran a file processing script through python:
Example like this:

import subprocess
for i in files:
   subprocess.run(<скрипт>)

The second time the server hung with an error

Too many open files in system.

.
The first time they did a reboot, but now it's night and it's a shame to mow like that. Tell me how to start a multiprocess correctly or how to limit it?
How long does it take for the garbage collector to close files? Will it close or reboot to request?
How to write so as not to repeat this mistake?
If you manage to enter the server, what to write to clean up your processes?

Answer 1

Limit the number of child processes

max_child_processes = 100
child_processes = []
for i, row in df.iterrows():
    while len(child_processes ) >= max_child_processes:
        sleep(0.1)
        child_processes = [p for p in child_processes if p.poll() is None]
    path = row.path
    subp = subprocess.Popen(f"echo {password} | sudo -S ./recog_wav.sh {path}",
                    stdin=subprocess.PIPE, stdout=subprocess.PIPE, shell=True)
    child_processes.appen(subp)