B
B
BogBel2016-01-28 01:58:51
Python
BogBel, 2016-01-28 01:58:51

How to generate Grab with own proxies?

There is a task to implement the work of asynchronous Grab::Spider through a proxy.
Now it looks like this
spider.py

from grab import Grab
from grab.spider import Spider, Task

class LinkSpider(Spider):

   def __init__(self, extrapage, homepage):
        super(LinkSpider, self).__init__()
        self.initial_urls = ['some_url']
        self.homepage = homepage
        self.matches = []
        self.OUdict = {}
   def task_initial(self, grab, task):
        do_smthng
        link = 'https://api.ipify.org/?format=json'
        g = Grab(url=link)
        yield Task('parser', grab=g, link=link)
   def task_parser(self,grab,task):
        response = grab.response.body
        print(response)

In this case, everything is done and my current ip is printed to the console.
Now add proxy-setup
from grab import Grab
from grab.spider import Spider, Task

class LinkSpider(Spider):

   def __init__(self, extrapage, homepage):
        super(LinkSpider, self).__init__()
        self.initial_urls = ['some_url']
        self.homepage = homepage
        self.matches = []
        self.OUdict = {}
   def task_initial(self, grab, task):
        do_smthng
        link = 'https://api.ipify.org/?format=json'
        g = Grab()
        g.setup(url=link, proxy="120.52.72.47:80", proxy_type="http")
        yield Task('parser', grab=g, link=link)
   def task_parser(self,grab,task):
        response = grab.response.body
        print(response)

I took proxies from different sites, for example:
So, in the case of such a launch, the first task is called, but it simply doesn’t get into the second one (that is, I don’t see any errors or ip-address in the console, after a couple of seconds the program ends.
Tell me, maybe I I missed something and somewhere it was necessary to make a global setup, did it as it is written in the docks, maybe there is another way, maybe you can do a global setup for all grab objects? elementary
Watson!

Answer the question

In order to leave comments, you need to log in

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question