How to parse several million domains?

Question

Domain name market

It is necessary to parse a decent number of sites and collect information from them (only muzzles), the problem is that most software is terribly, terribly slow plows. How can this problem be solved, a ninja-style parser or software with distributed parsing?

Answer 1

One-liner on bash + parallel + wget/curl
And install dns server locally

Answer 2

did stupidly on a node like that.
all *.RU domains (5.5 million) in a few days.

Answer 3

Python + Celery + Elasticsearch