What is the fastest way to check website availability?

Question

open source

There are 250 million domains. You need to check this entire list once a day. What is the ready software? There are 5 VDS with a gigabit channel.
You need to make sure that the main page is available and does not redirect anywhere

Answer 1

The task is either something invented out of my head, or suddenly of an enterprise order. Natural questions arise - where did these 250 million come from? Why check them all? Why do it daily?
I will join the previous speakers - you can do it in different ways, even curl in many threads. But the answers need to be added somewhere, the received data needs to be processed somehow, showing statistics, etc.
The short answer is: "there is ready-made software - anyone capable of making an HTTP request, having received a response code."
PS - 300th response codes do not indicate the unavailability of the domain, IMHO, they also need to be taken into account.

Answer 2

Isn't this a question at an interview in a bucket?
Roughly, 125M per node, 86k seconds per day, it turns out one and a half thousand hosts per second.
If we take the average response time of half a second, then we need from 700 parallel processes.
And who's spinning there - it doesn't matter, stupidly curl.
Well, the dispatcher who will manage all this.
Although I'm lying. what half a second. Well, we check the availability, that is, some part will run into a timeout => processes are much over a thousand. I'm afraid the VDS will burst.
In general, the answer is as usual - hire a specialist / service

Answer 3

It smacks of fraud)) Because today there are 220,895,309 domains in all zones.
Well, that's okay.
In general, I solved a similar problem. I will not give the code, because I abandoned this idea.
https://qna.habr.com/q/1023898