Answer the question
In order to leave comments, you need to log in
Why does the server not give the file via a direct link?
Hello! There is a site that I'm trying to parse and I have direct links to all the files that I need from this site and which were collected in advance (all graphic files - jpg). There is some kind of protection on the site, there is a specific number of files that can be downloaded from it, and if there is an excess, then just a void opens on these direct links. How can this be if I access direct links, what is now a graph on this link, and then emptiness? How to get around this limitation?
PS I don't have normal proxies, and everything I've tried is very slow, but nevertheless, everything works through them for some time and each limit you just need to change the proxy, but we are talking about a huge number of files and this method simply does not work.
How to bypass this protection? Help advice please!
Thank you!
Answer the question
In order to leave comments, you need to log in
The answer to the original question is easy. And thousands of ways from Web Application Firewall to NginX's built-in lua module.
Answering the question posted in the comments:
function getContent($url, $referer = null, $proxies = array(null))
{
$proxies = (array) $proxies;
$steps = count($proxies);
$step = 0;
$try = true;
while($try){
// create curl resource
$ch = curl_init();
$proxy = isset($proxies[$step]) ? $proxies[$step] : null;
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_REFERER, $referer);
curl_setopt($ch, CURLOPT_USERAGENT, "Opera/9.80 (Windows NT 5.1; U; ru) Presto/2.9.168 Version/11.51");
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); //return the transfer as a string
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
$output = curl_exec($ch); // get content
$http_code = curl_getinfo($ch, CURLINFO_HTTP_CODE); // Получаем HTTP-код
// close curl resource to free up system resources
curl_close($ch);
$step++;
$try = (($step < $steps) && ($http_code != 200));
}
return $output;
}
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question