K
K
khodos_dmitry2017-06-05 14:54:09
Parsing
khodos_dmitry, 2017-06-05 14:54:09

Why is parsing so slow?

Scab now site 2 million pages. On average, 10 pages are parsed per minute.
I use multithreading and proxy. Multithreading type several identical scripts with different page ranges. At what, that in one stream, that in several the speed is the same.
I use paid proxies.
The channel is not crowded, as you can watch full hd videos in parallel. When viewing, the speed neither increases nor decreases.
Enough RAM, processor too.
I'm using phpQuery, but I don't think it's the problem, since the processor is not clogged.

Answer the question

In order to leave comments, you need to log in

1 answer(s)
D
Dimonchik, 2017-06-05
@dimonchik2013

multucurl will save you, 200 per second is real

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question