I
I
iva36822015-08-23 19:16:29
PHP
iva3682, 2015-08-23 19:16:29

Why does an incomplete page come up when using curl + proxy?

HTTP experts, help with a problem!
I use curl (in PHP) through a proxy (no access to a proxy) to get the content of the site (HTTP proxy). Very often incomplete pages arrive, break off at the semi-tag (for example
...<p class="myclas, the content ends further). There are no regularities when and how the content will break. Moreover, if you do not receive through a proxy, there is no problem. I searched for solutions on the Internet, found the following solutions:
1) I updated the version of curl to 7.36 - it did not help.
2) Sending header "Expect:" - didn't help.
3) Timeout set to 0 - did not help.
4) Set the CURLOPT_HTTP_VERSION option to CURL_HTTP_VERSION_1_0 - if the response comes in HTTP 1.1 format, it still breaks.
5) Received content bypassing CURLOPT_RETURNTRANSFER = 1 via ob_start and so on - didn't help.
6) CURLOPT_HTTP_TRANSFER_DECODING = 0 and CURLOPT_HTTP_CONTENT_DECODING = 0 - did not help.
If you enable VERBOSE mode in curl, then on problem pages it says
...
nread <= 0, server closed connection, bailing
curl: (18) transfer closed with outstanding read data remaining
...
Also, the problem is observed only with Transfer-Encoding: chunked . Seen on many sites.
How do you still get the content completely, or, if this is not possible, how to determine that the content came incomplete?

Answer the question

In order to leave comments, you need to log in

1 answer(s)
D
Dimonchik, 2015-08-24
@iva3682

is your proxy controlled?
determine what is incomplete

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question