W
W
WebCab2014-10-22 15:36:19
PHP
WebCab, 2014-10-22 15:36:19

PHP How to make parsing with pause?

Good afternoon.
Today we thought about parsing car categories from one of the sites, for which we wrote a simple parser:

<?php 
function cURL_AutonavigatorRu($level = false, $model_id = false){
    #http://www.autonavigator.ru
    $ch = curl_init('http://www.autonavigator.ru/dispatcher.pl');
    curl_setopt($ch, CURLOPT_POST, 1);
    curl_setopt ($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/38.0.2125.104 Safari/537.36"); 
    $headers = array
    (
        'Accept: application/json',
        'Accept-Language: ru-RU,ru;q=0.8,en-US;q=0.6,en;q=0.4',
        'Accept-Encoding: gzip,deflate',
        'Accept-Charset: windows-1251,utf-8;q=0.7,*;q=0.7'
    ); 

    curl_setopt($ch, CURLOPT_HTTPHEADER,$headers); 
    curl_setopt($ch, CURLOPT_REFERER, "http://www.autonavigator.ru/my/offer_add/");

    if($level == '1'){
        curl_setopt($ch, CURLOPT_POSTFIELDS, 'class=list&method=make&show_all=1&vehicle=car&type=used');
    }
    elseif($level == '2' && $model_id){
        curl_setopt($ch, CURLOPT_POSTFIELDS, 'class=list&method=model&show_all=1&vehicle=car&type=used&make_id='.$model_id);
    }
    elseif($level == '3' && $model_id){
        curl_setopt($ch, CURLOPT_POSTFIELDS, 'class=list&method=modif&show_all=1&model_id='.$model_id);
    }
    else{
        curl_setopt($ch, CURLOPT_POSTFIELDS, 'class=list&method=modif&show_all=1&model_id='.$model_id);
    }

    curl_setopt($ch, CURLOPT_ENCODING , "gzip");
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);

    $result = curl_exec($ch);
    curl_close($ch);

    $json = json_decode(iconv("windows-1251","utf-8",$result), true);
    return $json['list'];
}


$ArrAuto = cURL_AutonavigatorRu('1');

foreach($ArrAuto as $auto) {

    echo $auto['value'].'<br>';


    //sleep(2);
    $AllModif1 = cURL_AutonavigatorRu('2',$auto["id"]);
    var_dump($AllModif1);
    echo '<br><br>----------------------------------<br><br>';

}

?>

However, there was a small problem, the parser receives modifications only for the first 5-7 cars, i.e. Probably there is a restriction on obtaining data on time.
Tell me how to get around this problem?

Answer the question

In order to leave comments, you need to log in

5 answer(s)
K
Kirill Firsov, 2014-10-22
@Isis

The question and the description of the question do not match.
For the first sleep, for the second set_time_limit

D
Dmitry Batulin, 2014-10-22
@DmitryPRG

set_time_limit( int $seconds );
more details at:
php.net/manual/ru/function.set-time-limit.php
This function does not work if PHP is running in safe mode.
The function must be specified 1 time at the very beginning of the script.

S
Sergey Prosvirnin, 2014-10-22
@sprosvirnin

sleep() or usleep()

A
Alexander Zelenin, 2014-10-22
@zelenin

guzzle+sleep+exception handling

T
Taras Serevann, 2014-10-26
@Taras_Serevann

sleep(time in ms.);

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question