K
K
khodos_dmitry2018-06-05 10:19:28
PHP
khodos_dmitry, 2018-06-05 10:19:28

Why does this script take all the memory?

<?php
  require_once 'phpQuery/phpQuery-onefile.php';
  $link = mysqli_connect('localhost', 'root', '', 'base2');
  ini_set('max_execution_time', 0);
  ini_set('memory_limit', '3048M');
  while (true) {
    $query = "SELECT * FROM `str` WHERE `html` IS NOT NULL AND `inserted` = 0 AND `url` LIKE '%/product/%' LIMIT 1000";
    $result = mysqli_query($link, $query) or die(mysqli_error($link));
    $data = mysqli_fetch_all($result, MYSQLI_ASSOC);
    if (!$data) break;
    foreach ($data as $dat) {
      insert_item($dat);
      $date = new DateTime();
      file_put_contents("end.txt", $dat["id"]."\t". $date->format('H:i:s:u') ."\r\n", FILE_APPEND);
    }
    mysqli_close($link);
  }
?>

8 GB of RAM eats up.
Function:
function insert_item($dat) {
    global $link;
    $pq = phpQuery::newDocument($dat['html']);
    $cats = $pq->find(".link_string");
    foreach ($cats as $cat) {
      $pq_cat = pq($cat);
      $icat[] = mysqli_real_escape_string($link, trim($pq_cat->text()));
    }
    $name = mysqli_real_escape_string($link, trim($pq->find("div.blockContentImgText > div.ContentProduktTextBlock > div.zagolovok_ProduktTextBlock")->text()));
    $params = $pq->find(".BlockTovarRama");
    foreach ($params as $param) {
      $pq_param = pq($param);
      $param_name[] = mysqli_real_escape_string($link, trim($pq_param->find(".BlockTovarHarakteristiki_l")->text()));
      $param_value[] = mysqli_real_escape_string($link, trim($pq_param->find(".BlockTovarHarakteristiki_r")->text()));
    }
    $price = mysqli_real_escape_string($link, trim($pq->find(".BlockTovar_textblock_zagolovok_tovar_prise")->text()));
    $descr = mysqli_real_escape_string($link, trim($pq->find(".text_kontent")->text()));
    $img_url = mysqli_real_escape_string($link, "http://bvb-alyans.ru".trim($pq->find(".blockProduktText")->attr("src")));
    $query = "INSERT INTO `item` SET `name` = '$name', `cat1` = '{$icat['1']}', `cat2` = '{$icat['2']}', `cat3` = '{$icat['3']}', `cat4` = '{$icat['4']}', `cat5` = '{$icat['5']}', `cat6` = '{$icat['6']}', `descr` = '$descr', `price` = '$price', `url_str` = '{$dat['url']}' ON DUPLICATE KEY UPDATE `url_str` = '{$dat['url']}', `descr` = '$descr', `price` = '$price'";
    mysqli_query($link, $query) or die(mysqli_error($link));
    $item_id = mysqli_insert_id($link);
    if ($item_id) {
      if ($img_url) {
        $query = "INSERT IGNORE INTO `img` SET `url` = '$img_url', `item_id` = $item_id";
        mysqli_query($link, $query) or die("Ошибка при добавлении картинки ".mysqli_error($link));
      }
      $count = count($param_name);
      for ($i = 0; $i < $count; $i++) {
        $query = "INSERT INTO `param` SET `name` = '{$param_name[$i]}', `value` = '{$param_value[$i]}', `item_id` = $item_id ON DUPLICATE KEY UPDATE `value` = '{$param_value[$i]}'";
        mysqli_query($link, $query) or die("Ошибка при добавлении параметра ".mysqli_error($link));
      }
    }
    $query = "UPDATE `str` SET `inserted` = 1 WHERE `id` = {$dat['id']}";
    mysqli_query($link, $query) or die(mysqli_error($link));
  }

Answer the question

In order to leave comments, you need to log in

3 answer(s)
V
Vitaly, 2018-06-05
@rim89

first look at mysql-slow
/var/log/mysql/mysql-slow.log if it's
clean there

D
Danbka, 2018-06-05
@Danbka

You have an endless loop

while (true) {
}

It won't end like this
$data = mysqli_fetch_all($result, MYSQLI_ASSOC);
    if (!$data) break;

because mysqli_fetch_all() false doesn't return: php.net/manual/en/mysqli-result.fetch-all.php

P
Paltos, 2018-06-05
@Paltinik

phpQuery, most likely, I just recently had problems with RAM with this library, all the memory was clogged in a large loop and the script crashed with an error.
Proof screen:
I tried without it, the consumption level remained the same before the start of the script..
try nokogiri nokogiri github is a very light library, but I had problems with encoding with it.
I found a DiDOM DiDOM github option for myself , a lot of options for working with it, it seems better than pq...

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question