How to speed up the import of a CSV file with 300k records?

Question

PHP

Hi all.
In general, such a situation, there is a CSV file with 300k lines.
I have a virtual hosting with a time limit of 300 seconds.

I read the file and update the price like this:

//если файл читается...
if (($handle = fopen("price.csv", "r")) !== FALSE) {
        //обрабатываем построчно...
  while (($data = fgetcsv($handle, 4000, ";")) !== FALSE) {
                //обходим инфоблоки товаров по артикулу....
    $arFields = "";
    $arSelect = Array("ID", "NAME", "DATE_ACTIVE_FROM");
    $arFilter = Array("IBLOCK_ID"=>5, "PROPERTY_CML2_ARTICLE" => $data[4]);
    $res = CIBlockElement::GetList(Array(), $arFilter, false, Array("nPageSize"=>2), $arSelect);
    while($ob = $res->GetNextElement())
    {
      $arFields = $ob->GetFields();

    }

    if ($arFields) {
      //если найден элемент обнавляем цену
                        CPrice::SetBasePrice($arFields['ID'], (int)$data[12], "RUB", 0);
      
    }else{
                        //если НЕ найден элемент выводим информацию по нему
      ?><pre><? print_r($data)?></pre><hr><?	
    }
  }
  echo $row;
  fclose($handle);
}

:

The problem is that when there are 5k lines, it works for a long time, but it takes less than 300 seconds,
but there are 300k positions in the file, this is 60 times more and it takes more time to process.

Now the question is: how, for example, to read from a certain line, or somehow, for example, I would run it on the crown every 10 minutes, reading 1000 lines and deleting them after reading ..., or is there some other way?

I tried to switch to VPS, but the VPS for 400 rubles per month turned out to be weaker than the virtual one for 300 rubles

Answer 1

if the file is simple, fields without quotes and line breaks, then fgets/explode
will be 40 times faster than fgetcsv
plus the whole process should be wrapped in a transaction. also every 70 times should speed up writing to the database
But in general, of course, you need to profile first. That is, to see which part of the script takes how long. not to point your finger at the sky.
Therefore
1. Put everything on your home computer.
2. add before the loop $time = time(); and after the loop - echo time() - $time;
3. First, comment out all work with the database - that is, leave only reading from the file, and see how long it takes
4. Uncomment the read statements from the database, and subtract the time to read the file from the result - we get how long it takes to read
5. Uncomment the rest and measure the full execution of the script, with a write to the database.
With these numbers already go here.
And ask a meaningful question, "how can I speed up such and such an operation", and not "how can I screw crooked crutches to my crooked code"

Answer 2

$res = CIBlockElement::GetList(Array(), $arFilter, false, Array("nPageSize"=>2), $arSelect);

Take it out of the loop and update only those prices that have changed and you will be happy.