Why does apache + nginx give 500 error when processing images?

Question

PHP

There is a php script that processes a large number of images (resize, apply watermarks, etc.). The problem is that the script returns a 500 error after 40 minutes of work, and then, with each subsequent launch, it falls off even earlier, although the timeout for executing the request is about an hour. Logs upstream prematurely closed connection while reading response header from upstream
Apache
max_execution_time 3600
max_input_time 3600
upload_max_filesize 512M
post_max_size 512M
memory_limit 512M
Nginx
proxy_connect_timeout 3600s;
proxy_send_timeout 3600s;
proxy_read_timeout 3600s;
proxy_buffer_size 512k;
proxy_buffers 512 4096k;
proxy_busy_buffers_size 4096k;
proxy_temp_file_write_size 4096k;
client_max_body_size 30m;
client_body_buffer_size 4096k;
large_client_header_buffers 8 32k;
The script itself clears the disk cache after processing each directory.
How can I solve this problem without resorting to cron and / or running the script directly on the server?

Answer 1

The entry in the nginx logs “upstream prematurely closed connection while reading response header from upstream” means exactly the premature closing of the connection of the proxied server, that is, an error during script execution. See the logs of the script itself or Apache. Either the script lacks resources, or some libraries/modules, or the code itself contains errors.
By the way, the status code 500 (Internal server error) hints at an error. If the upstream server does not respond for too long, a 504 (Gateway timeout) status code is returned.

Answer 2

in general, for such resource-intensive tasks, the execution time of which is unpredictable, it is desirable to use queues. A very simple queue server - gearman.org more
material here

FcgidMaxRequestLen 512000000
<IfModule mod_fcgid.c>
    FcgidBusyTimeout 3600
    FcgidIOTimeout   3600
</IfModule>

Answer 3

There is php-cli for such tasks.