D
D
Dmitriy2019-08-25 23:14:30
PHP
Dmitriy, 2019-08-25 23:14:30

Bans when parsing (too many requests). How to fix?

<?php
  require_once('../phpQuery/phpQuery.php');
  require_once('../curl.php');
  require_once('../simple_html_dom.php');



  $curl = new Curl();

  $res = $curl->get('http://www.bestchange.ru/qiwi-to-ripple.html');
  $doc = phpQuery::newDocument($res->body);
  $changes = $doc->find('div#rates_block table#content_table tr');

  foreach ($changes as $change) {
    $pq = pq($change);
  }
?>
How to make sure that the site is not banned when parsing?
Too many requests.
Scab exchangers with bestchange.ru.
What does banned mean?
Uploaded the parser to the hosting.
Scab and output exchangers in the form of a table.
At first, everything outputs well, and after a few minutes, nothing outputs.

Answer the question

In order to leave comments, you need to log in

6 answer(s)
R
Ronald McDonald, 2019-08-25
@Zoominger

Use proxies for burst requests, for example.

D
Dmitriy, 2019-08-26
@dmitriyuvin

Added CURL.

$ch = curl_init('http://www.bestchange.ru/qiwi-to-zcash.html');
  curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
  curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
  curl_setopt($ch, CURLOPT_PROXY, '188.130.134.13:46546'); // есть список proxy
  curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_HTTP);
  curl_close($ch);

S
Sergey Gornostaev, 2019-08-26
@sergey-gornostaev

How to parse without a ban?

G
grinat, 2019-08-26
@grinat

Do through this server https://apify.com/store?type=acts

E
Eugene, 2019-08-25
@bratok191

Write what kind of site at least?
and in what way does it ban? maybe, for example, it gives a captcha for solving - in this case, you can use the rucaptcha recognition services.
or change ip if it doesn't help...
+ you can reduce the speed of parsing - then it might be better too.

R
rasschitai, 2020-03-22
@rasschitai

tin) well, there is an api there that gives the zip file ... one said about the proxy, and away we go ...
5e7720f5e9921816245926.png

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question