Is there an alternative to regular TCP sockets for a multithreaded parser?

Question

ruby

Task: collect information about ~100 million pages on different domains as quickly as possible using the HEAD method.
Now the parser is written in ruby, uses HTTPClient and multithreading.
Maybe there are faster libraries for such tasks?

Answer 1

1) Run on jruby/rubinius to get real multi-threading for distributing threads across cores, otherwise the GIL interferes
2) Run multiple processes

Answer 2

Well, let's start with the fact that HTTP uses TCP as a transport (at least from the server side), so ... no.