T
T
Timur Evgazhukov2017-04-07 09:30:35
Java
Timur Evgazhukov, 2017-04-07 09:30:35

How to organize http requests from java using various anonymous proxies?

I made a parser in java that collects data from a specific site, but unfortunately, after some time it starts to require captcha input, moreover, it blocks at the beginning and only displays this captcha after a few hours. Slowing down requests (for example, once every few seconds) does not help. Perhaps it will help in such cases to work through various proxies and anonymizers, but so far I don’t really know what services to work through and how to program it in java.

Answer the question

In order to leave comments, you need to log in

2 answer(s)
D
Dmitry Alexandrov, 2017-04-07
@evgajukov

Take a list of "different anonymous" proxies. Further simply

Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("ip прокси", ПОРТ_ПРОКСИ));
conn = new URL(urlString).openConnection(proxy);

A
al_gon, 2017-04-07
@al_gon

Everything is as Dmitry Alexandrov
said. The only thing is do not use the standard library, it is more than bugs.
Take https://github.com/google/google-http-java-client , it is also not without bugs, but there are many times fewer of them.
The initialization is the same:

Builder builder = new NetHttpTransport.Builder();

               try {
      if (doNotValidateCertificate) {
        builder.doNotValidateCertificate();
        builder.setHostnameVerifier(null);
      }
    } catch (final GeneralSecurityException e) {
      throw new IllegalStateException(e);
    }

    if (useProxy) {
      final Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(REV_PROXIES_ONLINE_HOST,
                                      REV_PROXIES_ONLINE_PORT));
      builder = builder.setProxy(proxy);
    }

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question