Answer the question
In order to leave comments, you need to log in
Why does this script take all the memory?
<?php
require_once 'phpQuery/phpQuery-onefile.php';
$link = mysqli_connect('localhost', 'root', '', 'base2');
ini_set('max_execution_time', 0);
ini_set('memory_limit', '3048M');
while (true) {
$query = "SELECT * FROM `str` WHERE `html` IS NOT NULL AND `inserted` = 0 AND `url` LIKE '%/product/%' LIMIT 1000";
$result = mysqli_query($link, $query) or die(mysqli_error($link));
$data = mysqli_fetch_all($result, MYSQLI_ASSOC);
if (!$data) break;
foreach ($data as $dat) {
insert_item($dat);
$date = new DateTime();
file_put_contents("end.txt", $dat["id"]."\t". $date->format('H:i:s:u') ."\r\n", FILE_APPEND);
}
mysqli_close($link);
}
?>
function insert_item($dat) {
global $link;
$pq = phpQuery::newDocument($dat['html']);
$cats = $pq->find(".link_string");
foreach ($cats as $cat) {
$pq_cat = pq($cat);
$icat[] = mysqli_real_escape_string($link, trim($pq_cat->text()));
}
$name = mysqli_real_escape_string($link, trim($pq->find("div.blockContentImgText > div.ContentProduktTextBlock > div.zagolovok_ProduktTextBlock")->text()));
$params = $pq->find(".BlockTovarRama");
foreach ($params as $param) {
$pq_param = pq($param);
$param_name[] = mysqli_real_escape_string($link, trim($pq_param->find(".BlockTovarHarakteristiki_l")->text()));
$param_value[] = mysqli_real_escape_string($link, trim($pq_param->find(".BlockTovarHarakteristiki_r")->text()));
}
$price = mysqli_real_escape_string($link, trim($pq->find(".BlockTovar_textblock_zagolovok_tovar_prise")->text()));
$descr = mysqli_real_escape_string($link, trim($pq->find(".text_kontent")->text()));
$img_url = mysqli_real_escape_string($link, "http://bvb-alyans.ru".trim($pq->find(".blockProduktText")->attr("src")));
$query = "INSERT INTO `item` SET `name` = '$name', `cat1` = '{$icat['1']}', `cat2` = '{$icat['2']}', `cat3` = '{$icat['3']}', `cat4` = '{$icat['4']}', `cat5` = '{$icat['5']}', `cat6` = '{$icat['6']}', `descr` = '$descr', `price` = '$price', `url_str` = '{$dat['url']}' ON DUPLICATE KEY UPDATE `url_str` = '{$dat['url']}', `descr` = '$descr', `price` = '$price'";
mysqli_query($link, $query) or die(mysqli_error($link));
$item_id = mysqli_insert_id($link);
if ($item_id) {
if ($img_url) {
$query = "INSERT IGNORE INTO `img` SET `url` = '$img_url', `item_id` = $item_id";
mysqli_query($link, $query) or die("Ошибка при добавлении картинки ".mysqli_error($link));
}
$count = count($param_name);
for ($i = 0; $i < $count; $i++) {
$query = "INSERT INTO `param` SET `name` = '{$param_name[$i]}', `value` = '{$param_value[$i]}', `item_id` = $item_id ON DUPLICATE KEY UPDATE `value` = '{$param_value[$i]}'";
mysqli_query($link, $query) or die("Ошибка при добавлении параметра ".mysqli_error($link));
}
}
$query = "UPDATE `str` SET `inserted` = 1 WHERE `id` = {$dat['id']}";
mysqli_query($link, $query) or die(mysqli_error($link));
}
Answer the question
In order to leave comments, you need to log in
first look at mysql-slow
/var/log/mysql/mysql-slow.log if it's
clean there
You have an endless loop
while (true) {
}
$data = mysqli_fetch_all($result, MYSQLI_ASSOC);
if (!$data) break;
phpQuery, most likely, I just recently had problems with RAM with this library, all the memory was clogged in a large loop and the script crashed with an error.
Proof screen:
I tried without it, the consumption level remained the same before the start of the script..
try nokogiri nokogiri github is a very light library, but I had problems with encoding with it.
I found a DiDOM DiDOM github option for myself , a lot of options for working with it, it seems better than pq...
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question