H
H
hrvasiliy2015-06-26 17:11:20
PHP
hrvasiliy, 2015-06-26 17:11:20

What am I doing wrong when parsing?

There is a donor site, I'm trying to take information from it and write it to a file. (Days 10-15 ago everything was perfectly parsed with the help of CURL). Now, information is written to the file that it is required to include JS, I change the method for obtaining the page from CURL'a to PhantomJS and gives the same error. I don’t understand, did the donor site strengthen the protection? How then to get information?

function curl($url, $userAgent = null, $proxy = null, $referer = null)
{
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_USERAGENT, $userAgent);
curl_setopt($curl, CURLOPT_HEADER, 0);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_REFERER, $referer);
curl_setopt($curl, CURLOPT_PROXY, $proxy);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, 1);
$out = curl_exec($curl);
curl_close($curl);
return $out;
}

Answer the question

In order to leave comments, you need to log in

3 answer(s)
S
Stalker_RED, 2015-06-26
@hrvasiliy

site-donor strengthened the protection?
Probably. Log in from a clean browser (private mode will help you), see what cookies are being created, add them to the CURL request.

A
Alexander Taratin, 2015-06-26
@Taraflex

Having activated the maximum level of telepathy, I will assume the following.
The donor site has switched to ajax and main data loading via js.
Users without js are given a stub asking them to enable js, which is what you see in the CURL response.
Look at what ajax requests the donor page sends to the server and try to play them through curl already on your server.

A
Alexander Melekhovets, 2015-06-26
@Blast

Please contact the developers of the "donor" site with this question. Ask for advice, ask about the availability of an API or the ability to unload the database. Ideally, tell them how to find you in the real world, then there is a high probability that they will do everything themselves.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question