Why is there a lot of RAM consumption in background PHP scripts?

Question

PHP

There are 300 PHP scripts running in the background. In different places of the script, the memory consumption log is stuffed

$mem = round((memory_get_usage() / 1024000), 3) . ' MB';

As a result, all values ​​give out from 5 to 6 mb. Those. it turns out 300 scripts * 6 MB of memory = 1,800 MB of consumed RAM.
But the free -m command shows that 12-13 GB is used.
There are suspicions that this is due to MySQL. each instance of the script pulls a connection to the database that hangs for 5-7 hours (script execution time).
SHOW PROCESSLIST
shows just over 300 processes.
There is an option to refuse a permanent connection to the base, write an API that will work with this base. Those. the background php script (of which there are 300 pieces) will not work directly from the database, but via the REST API. This will avoid a large number of hanging connections to the database.
Will this help reduce RAM consumption or maybe this is not the reason?

Answer 1

As for me, 300 background pieces is a bit much. Do you need so much in realtime?
All non-rilltime requests to the background script - allocate, queue and / or transfer to rest server / service execution. Let the background script / service / server (in the singular or in a small number) execute them slowly pulling them out of the queue.
PS Simply by transferring the functionality of the background script to the rest api, nothing will change. There will also be a number of connections. And "the same rake, only in profile."

Answer 2

Are your background processes mainly writing or reading?
Requests for the rest API will either generate new connections each time, or, if you are accessing a daemon, they can interfere with each other and line up in a queue.
Maybe use something like gearman ?

Answer 3

from the client side, an open database connection is just an open socket. so that's hardly a problem.
why 300 "hanging for 5-7 hours" scripts? Are they working hard at this time?

Answer 4

if processes fork, then memory is rummaged

Answer 5

Ok, let's say 32 GB is enough for me for about 900-1000 processes (tasks). I'm fine with that, but there's a new problem.
A little time passed and with the same number of tasks, memory began to devour 2-2.5 times more.
I look at htop and see that mysql has not one process hanging as before, but many, each of which takes about 300 MB of memory.

Answer 6

I assume that 5-6 MB per process is not so much, this memory can simply be occupied by the loaded libraries and modules of your application. To test this hypothesis, create a script that loads the same libraries as your background scripts, but does not fetch anything from the database, and measure the memory usage. But, if the scripts start using more memory over time, then you may not clear some variables, for example, there is an array into which some data is added at each iteration, but not cleared at the end. In that case, I think the problem is in your code.