How to speed up the running time of WP All import?

Question

Server optimization

There is an online store for 12,000 products.
Products are listed in a csv file, which is downloaded from the link and updated every 30-60 minutes.
Images of goods are set by links to pictures and are loaded by the link. One picture weighs 100-120Kb.

But when importing, such a picture is loaded for 2 minutes. Because of what, updating goods turns into an adventure that lasts more than one cat's life.

Attempts to launch via cron or as indicated here https://ivannikitin.com/2017/03/25/wp-all-import-cli/
do not bring significant results.

We tried to check the response of the server from which we pull the pictures, everything is fine there too. We tried to download pictures via wget, 33000 images are downloaded instantly and are downloaded in 2 hours.

BUT uploading to the server is also not a special option. The speed of creating one product becomes around a minute. And downloading pictures, uploading them to the server is not quite the right decision, because the goods in the file are updated, for a week about + 250-300 positions. Therefore, we chose WP All Import, because. it allows you to re-download with the same settings and set up automatic launch of updates.

Those who provide hosting services can't give a clear answer except "I need more fine-tuning of the server." But it is not entirely clear what exactly can be configured there.

I would be very grateful for the tips, the more the question arises that the number of goods is relatively small.

A few words about the server
Operating System Ubuntu 18.04 64bit Minimal
Disk Space 180 GB
Bandwidth 120 TB
Memory 8 GB
8 cores

PS are facing 504 errors - time waiting for a response from the web server, as well as a Cannot allocate memory error during the import process. Attempts to increase the php limits do not resolve the error.

Answer 1

Firstly, All import itself is not fast, it is a price for universality.
Secondly, a bunch of actions and hooks are triggered during improte, they can be disabled by the appropriate setting, but it’s not worth it at all, when importing pictures, auto-generation of their various sizes is triggered, and in general there is a whole chain of events.
To speed up, in your case, you need to carry out a bunch of activities
- cleaning and optimizing the database
- refactoring the theme
- refactoring the called hooks
- switching to auto-generation of images on the fly instead of the built-in
one - optimizing the upload size
And a lot more.
Your server is normal, its head is enough. Well, in general, a 2-minute picture, it's scary to think what's going on under your hood, you need to deal with it.

Answer 2

Hello!
To solve the problem, consider the options below.
1. Set limits on PHP.
Although you have already set the limits, you should take into account the fact that there are several versions of PHP on the server.
PHP version native on the server can be one, and the version for the site can be different.
How to set limits.
There are many ways to set limits. Below are a few of them:
1. Through the .user.ini file in the root of the site.
2. Through the .htaccess file in the root of the site.
3. On the server globally. Examples: /etc/php.ini /opt/php73/etc/php.ini
4. In the control panel in the PHP settings.
Set limits and check that they are set.
For check:
- Create a file in the root of the site php-ver.php
- Enter the code below.
<?php
phpinfo()
- Open the site xn--_-7sbbf2b7bj7b/php-ver.php
- Check that the limits you set are set correctly.
max_execution_time = 90
memory_limit = 300M
post_max_size = 300M
upload_max_filesize = 300M
2. Configure limits on the web server:
- Apache server:
Most often in /etc/httpd/conf/httpd.conf file
Parameter:
Timeout 7200 (7200 seconds = 2 hours, set your value).
- Nginx server
Most often in the file /etc/nginx/nginx.conf !http section!
proxy_read_timeout 7200;
proxy_connect_timeout 300;
client_body_buffer_size 1024M;
3. mysql limits
key_buffer_size = 512M
max_allowed_packet = 24M
query_cache_size=1024M
join_buffer_size=1024M
tmp_table_size=1024M
max_heap_table_size=1024M
The limits are very high, but you should have enough. Experiment with them.
Follow the above solutions and the issue should be resolved.

Answer 3

Get in touch if you don't understand