How to use all the resources of the Internet connection when parsing sites?

Question

PHP

I wrote a parser program in PHP, it works on Denver. I began to be dissatisfied with the speed and, for comparison, I tested the operation of this program with a friend. Its speed is up to 25Mbps, downloading torrents goes at a speed of 7 megabytes, against mine up to 6144 Kbps and downloading a torrent of 700 Kilobytes.
As a result, the difference in the speed of work turned out to be imperceptible, which surprised me very much.
Tell me how to use the Internet connection to its fullest?

Answer 1

If your script does not fully utilize the Internet connection, there are several probable reasons for this:
a) the site you are parsing sends a stream for 1 connection less than your channel
b) there are downtime in the script for parsing, working with the database, etc.
The solution is generally the same: running several copies of the parser at the same time, looking at the load of the channel and the processor - it’s hard to guess what you will run into first.
PS I can also advise you not to use fancy dom parsers and curl wrappers - simple curl and simple self-made regexp work 10-100 times faster.
PPS You are confused in the speed readings) 25Mbps is rounded ~ 3 megabytes / second, torrents cannot download at 7Mb / s