L
L
Lelouch2020-06-13 22:18:57
PHP
Lelouch, 2020-06-13 22:18:57

What is the best way to configure ngnx + php-fpm to run heavy scripts?

There are two weak virtual machines on AWS (1 CPU, 1 GB RAM). One runs nginx with php-fpm, the other runs MySQL.
Both VMs for the email service, with a database of approximately 500 thousand addresses.

Quite heavy scripts are executed by cron:
1. Actually sending (~10 emails per second).
2. Update segments.
3. Database import (rarely, but it loads a lot, importing a couple of hundred thousand can take several hours, slowing down everything else).

In addition, in the process of sending mailing lists to nginx and php-fpm (and further MySQL), a bunch of light requests arrive when users open emails and follow links (tracking of opens and links is configured). Well, plus all sorts of little things, like complaints about spam, unsubscribes, etc.

The statics for letters (pictures, fonts) have already been transferred to another server.

Problems:
1. When a mailing list is in progress, VMs obviously cannot withstand the load, clicks on links in emails can be processed for 10-15 seconds, which is a lot, and has a bad effect on clicks from the mailing list.
2. Updating segments is very slow. A simple segment is updated in about a minute. Heavy about 12 minutes. There are several dozen segments in total, most of which are heavy. It is advisable to update them at least several times a day.

Actually, I am not strong in administration and after reading a number of guides I wrote the following in the configs:

/etc/php/7.4/fpm/pool.d/www.conf

pm = ondemand
pm.max_children = 128
pm.start_servers = 2
pm.min_spare_servers = 2
pm.max_spare_servers = 4
pm.process_idle_timeout = 30s;
pm.max_requests = 2048

request_terminate_timeout = 0


/etc/php/7.4/fpm/php.ini
max_input_vars = 2048
memory_limit = 512M
max_input_time = 600
max_execution_time = 900


max_execution_time = 900 because cron scripts run every 15 minutes. Maybe 0 is better, like request_terminate_timeout?

The rest in php are default values.

/etc/nginx/nginx.conf
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
        worker_connections 2048;
        multi_accept on;
}

http {

        ##
        # Basic Settings
        ##

        charset utf-8;
        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;

        keepalive_requests 2048;
        keepalive_timeout 360;

        send_timeout 120;

        client_body_timeout 90;
        client_header_timeout 90;
        client_body_buffer_size 16M;
        client_header_buffer_size 8K;
        client_max_body_size 64M;
        types_hash_max_size 4096;
        reset_timedout_connection on;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # SSL Settings
        ##

        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_prefer_server_ciphers on;

        ##
        # Logging Settings
        ##

        access_log off;
        error_log /var/log/nginx/error.log crit;

        ##
        # Gzip Settings
        ##

        gzip on;
        gzip_vary on;
        gzip_proxied any;
        gzip_comp_level 6;
        gzip_buffers 16 8k;
        gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript image/svg+xml;
        gzip_min_length 256;

        ##
        # Virtual Host Configs
        ##

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}


In host config:
...
    location / {
        try_files $uri $uri/ /$uri.php$is_args$args;
    }

    location ~ \.php$ {
        try_files $uri =404;
        fastcgi_pass unix:/run/php/php7.4-fpm.sock;
        include fastcgi_params;
        fastcgi_read_timeout 900;
        fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
        fastcgi_param SCRIPT_NAME $fastcgi_script_name;
        fastcgi_intercept_errors on;
        fastcgi_index index.php;
    }
...


I would be grateful for advice on what is better to change (and why) or what to read in order to better understand how it all works. :)

PS: Reducing the sending speed is not an option, mailing to the full database goes on for 10-15 hours, which is already a very long time (letters dropped into the mailbox at night show the percentage of openings several times less than during the day).

Answer the question

In order to leave comments, you need to log in

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question