L
L
Lenn2020-11-03 03:48:36
PHP
Lenn, 2020-11-03 03:48:36

Error 429, parsing the site citilink.ru?

Good day!

$curl = curl_init($url);  #https://citilink.ru/search/?text=productId
    curl_setopt($curl, CURLOPT_HEADER, true);
    curl_setopt($curl, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:35.0) Gecko/20100101 Firefox/35.0");
    curl_setopt($curl, CURLOPT_COOKIE, '_pcl=eW4AAAAAnkTeOw==');
    curl_setopt($curl, CURLOPT_HTTPHEADER, array(
        'Content-Type: text/html'
    ));
    
    curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
    curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);

    $result = curl_exec($curl);
    
    return $result;


When I try to parse a page with a product from the citilink.ru site, after several successful requests, I get a 429 error with a constant reload, the error persists even when I request it again after a few hours, the response header is:

HTTP/1.1 429 Too Many Requests Server: QRATOR Date: Mon, 02 Nov 2020 23:42:17 GMT Content-Type: text/html; charset=utf-8 Content-Length: 993 Connection: keep-alive Keep-Alive: timeout=15 x-envoy-upstream-service-time: 13


in response body:
document.addEventListener("DOMContentLoaded", function() {
        var value = '';       
        var value2 = 'eW4A';
        value = value + value2;        
        var value2 = 'AAAA';      
        value = value + value2;        
        var value2 = 'nkTe';
        value = value + value2;         
        var value2 = 'Ow==';
        value = value + value2;
        var config = document.getElementById('el10998').dataset;
        Cookies.set(config.name, value, {
          domain: config.domain,
          expires: new Date(new Date().getTime() + 60 * 60 * 1000),
        });
        document.location.reload()
      });
    <div id="el10998" data-name="_pcl" data-domain=".citilink.ru"></div>


The reason for the constant reboot is clear:
document.location.reload()

But how to build a request correctly, what would the server accept...?

Answer the question

In order to leave comments, you need to log in

1 answer(s)
S
Sergey Gornostaev, 2020-11-03
@sergey-gornostaev

How to parse without a ban?

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question