Z
Z
zozo302015-05-07 13:08:01
ruby
zozo30, 2015-05-07 13:08:01

How to organize regular bypass of several sites through a proxy from ruby?

There is a need to regularly retrieve some data from the web interface of several third-party systems. (Prices for displaying advertising to optimize the profit from its display). Those. there is a possibility of some kind of sanctions from these systems for regular calls, you need to use various kinds of proxies, which, in which case, could be easily changed. The system is written in ruby, so the "robots" themselves are also in it. Currently using the socksify gem and a socks server using normal ssh (ssh -D 9999 -N server.domain.com). Robots use Net::HTTP::SOCKSProxy to work through this proxy.

However, this method has a number of problems.

  • regularly from the side of robots the error "SOCKS version is not 5" gets out. Googling does not give sane solutions
  • at some point, ssh-socks stops working, throws out to the console for each request "channel 1018: open failed: administratively prohibited: open failed". Restart helps, but up to a certain point.
  • regular timeouts of various kinds, but this may be due to problems in the systems themselves, from where we take the data, so the problem is secondary.


Thanks for the tips on which way to look.

Answer the question

In order to leave comments, you need to log in

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question