How to bypass DNS-Timeout with a huge number of DNS queries?

Question

Java

I'm calling the following code in parallel:

InetAddress address = Address.getByName(domain);
String ip = address.getHostAddress();

But after 500-600 requests DNS Timeout occurs. How to bypass it?

Answer 1

You can not bring the number of parallel tasks to infinity. This applies not only to computational tasks (which is obvious), but also to input-output (not so obvious, nevertheless, a fact). You need to somehow limit the parallelism. Lots of solutions. One of the best options for even load distribution, for example:

1. An array of fixed size threads is created (not a ThreadPoolExecutor).
   
2. All tasks are placed in the input ConcurrentLinkedQueue (or BlockingQueue, if the tasks arrive gradually and the next one has to wait).
   
3. Each thread gets the next task from the queue (completes if the queue is empty (for the case of ConcurrentLinkedQueue)), executes the task, puts the result into the output BlockingQueue.
   
4. The calling thread in the loop does take() from the output queue as many times as it put tasks into the input queue, after the end of the loop - processing is over.
   

The number of parallel streams is chosen empirically (depending on the channel width, in case of network I/O).
Advantages of this approach: at each moment of time (except for the completion period, when the input queue is already empty) exactly N tasks are executed, regardless of the duration of each individual task (the duration of long and short tasks can differ by tens/hundreds of times).