Answer the question
In order to leave comments, you need to log in
How to organize http requests from java using various anonymous proxies?
I made a parser in java that collects data from a specific site, but unfortunately, after some time it starts to require captcha input, moreover, it blocks at the beginning and only displays this captcha after a few hours. Slowing down requests (for example, once every few seconds) does not help. Perhaps it will help in such cases to work through various proxies and anonymizers, but so far I don’t really know what services to work through and how to program it in java.
Answer the question
In order to leave comments, you need to log in
Take a list of "different anonymous" proxies. Further simply
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("ip прокси", ПОРТ_ПРОКСИ));
conn = new URL(urlString).openConnection(proxy);
Everything is as Dmitry Alexandrov
said.
The only thing is do not use the standard library, it is more than bugs.
Take https://github.com/google/google-http-java-client , it is also not without bugs, but there are many times fewer of them.
The initialization is the same:
Builder builder = new NetHttpTransport.Builder();
try {
if (doNotValidateCertificate) {
builder.doNotValidateCertificate();
builder.setHostnameVerifier(null);
}
} catch (final GeneralSecurityException e) {
throw new IllegalStateException(e);
}
if (useProxy) {
final Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress(REV_PROXIES_ONLINE_HOST,
REV_PROXIES_ONLINE_PORT));
builder = builder.setProxy(proxy);
}
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question