The site does not load while working out the php script, how to solve the problem?

Question

PHP

Colleagues, hello everyone!
There is a question. There is a site whose directory is filled through data parsing through the framework directly from another site. Considering that the script has to open more than 3000 pages and collect data from them, it works out for a very long time. The site hangs while the script is running.
Tell me, is there any way, for example, to allocate only part of the server resources to the script? So that the site can load while the script is running? Or is the only option to just run the script on a separate server?

Answer 1

do not run the script from the site. Run as a scheduler

Answer 2

Separate the parser launch into a separate process with a low priority php.net/manual/en/function.proc-nice.php

Answer 3

Or is the only option to just run the script on a separate server?

Not the only way, but the only correct one.

have to open more than 3000 pages

Doesn't he open them at the same time? Do sleep between requests. Do not try to parse everything in one pass, load the "raw" html into a temporary table, and start parsing on this temporary data separately.
Ideally, do it through a RabbitMQ queue on a separate server. Here is a very good example .

Answer 4

What does "site is down" mean?
If the problem is that the login hangs (sessions in the database) - the session capture error hangs only for the user who launched the script - since the sessions are stored in the database. Store in files will not be a problem.
If the problem is that there are not enough resources - you need to run the cron script with nice (if the problem is in the processor) and ionice (if the problem is in the disks)
a typical script launch with a minimum priority:
0 3 * * * nice -n 19 ionice -c2 -n7 /usr/bin/php -f /home/bitrix/www/bitrix/cron/test/opt_price_gen.php" > /home/bitrix/www/bitrix/cron/test/logs/opt_price_gen.log

Answer 5

I would also like to add to lolhunter 's answer :
There is another problem that when parsing is too intensive work with the database and the database server hangs. There may be several reasons for this:
1. Just an intensive write - make a slip between records, combine many records into one transaction.
2 During the write, some data is retrieved from the database - this is such a ridiculous mistake that it almost always occurs. The code looks something like this:

foreach ($arElementsForUpdate as $arElm) {
// а тут какой-нибудь GetList с извлечением скажем цветов товаров из ИБ-справочника
// тут запись
}

Once, by removing such jambs from the cycle, I accelerated the work of the import script from 2.5 hours to 17 seconds. And everything stopped hanging.