L
L
Lorem Ipsum2014-12-25 18:25:11
Parsing
Lorem Ipsum, 2014-12-25 18:25:11

Does Google public DNS have limits?

Good evening.
I need to parse information from about 3 million sites, literally go to the site and download several pages (index and a couple of internal ones).
So, since I want to do this multithreaded, my script for some reason (definitely not because of the code, because everything is 100% OK with the code) is very slow. I think it's because of Google Public DNS limits. What are the options for solving the parsing problem in this case? (A proxy is also an option, but I want to take a back seat to this option, as it is completely impractical).

Answer the question

In order to leave comments, you need to log in

2 answer(s)
A
Armenian Radio, 2014-12-25
@gbg

Try to make a couple of DNS requests while the script is running.
Surely the problem is still in the code, only you do not know about it.
Use profiling.

O
oia, 2014-12-25
@oia

use several dns servers + root + 77.88.8.8 yandex

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question