Why is the page not loading with CURL?

Question

URL Handling

When the site loads normally in the browser, the page loads without problems.

When I make cURLa request, I bodydon’t get it at all, I only get head, while the page is updated automatically, I didn’t find any redirects.

I opened the inspector in the browser and found this line in the head:

<script type="text/javascript" async="" src="https://www.gstatic.com/recaptcha/releases/rCr6***********/recaptcha__ru.js" crossorigin="anonymous" integrity="sha384-uaB+28Q7G2Ccdbu4hmA0lq6Rp8c*******N0CUx+3B2UM"></script>

It turns out that this is some kind of protection against cURL?) Are there options to fix it or should I look for the problem elsewhere?

Answer 1

In order to avoid problems when loading a resource, it makes sense to imitate a real browser. It does this by passing the same headers that Google Chrome sends when requested. Example of passing headers:

<?php

// Указываем тип документа и кодировку:
header('Content-Type: text/html; charset=utf-8');

// Включаем отображение ошибок:

ini_set('error_reporting', E_ALL);
ini_set('display_errors', 1);
ini_set('display_startup_errors', 1);

// Адрес:
$url = 'https://ссылка_на_страницу';

// Создаём новый сеанс:
$curl = curl_init();

// Указываем адрес целевой страницы:
curl_setopt($curl, CURLOPT_URL, $url);

// О отключаем проверку SSL сертификата:
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, 0);

// Устанавливаем заголовки для имитации браузера:

$headers = [];
$headers[] = 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9';
$headers[] = 'Accept-Encoding: identity';
$headers[] = 'Accept-Language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7';
$headers[] = 'Cache-Control: no-cache';
$headers[] = 'Connection: keep-alive';
$headers[] = 'Host: ' . parse_url($url)['host'];
$headers[] = 'Pragma: no-cache';
$headers[] = 'Sec-Fetch-Dest: document';
$headers[] = 'Sec-Fetch-Mode: navigate';
$headers[] = 'Sec-Fetch-Site: none';
$headers[] = 'Sec-Fetch-User: ?1';
$headers[] = 'Upgrade-Insecure-Requests: 1';
$headers[] = 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.193 Safari/537.36';


curl_setopt($curl, CURLOPT_HTTPHEADER, $headers);

// Разрешаем переадресацию:
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, 1);

// Запрещаем прямяой вывод результата запроса:
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);

// Делаем сам запрос:
$result = curl_exec($curl);

// Завершаем сеанс:
curl_close($curl);

// Смотрм результат:
echo $result;