J
J
JRazor2014-07-02 17:59:48
Python
JRazor, 2014-07-02 17:59:48

Scrapy & PyQt4: how to send start to Scrapy from GUI and receive responses in GUI?

Hello fellow toasters.
I was developing a GUI for Scrapy and got confused: in the GUI, you need to load the module with Scrapy in order to run it with parameters, and in Scrapy, import the GUI in order to send information to QLabel and the second window. But when importing
To do this, I import a function from Scrapy and a class from the GUI, but I get:

File "C:\Users\Eugene\Scrapy\Majestic\gui\gui_for_scrapy.py", line 5, in <module>
    from spider_find_domains import run
  File "C:\Users\Eugene\Scrapy\Majestic\gui\spider_find_domains.py", line 16, in <module>
    from gui_for_scrapy import MyWindow
  File "C:\Users\Eugene\Scrapy\Majestic\gui\gui_for_scrapy.py", line 5, in <module>
    from spider_find_domains import run
ImportError: cannot import name run

In MyWindow - loading 'form.ui' and working with it, and in 'run' - the usual launch of the Scrapy spider:
def run():
        options = {
            'CONCURRENT_ITEMS': concurrent_items,
            'USER_AGENT': 'Googlebot/2.1 (+http://www.google.com/bot.html)',
            'DOWNLOAD_DELAY': download_delay,
            'CONCURRENT_REQUESTS': concurrent_requests,
            'HTTPERROR_ALLOW_ALL': True,
            'SPIDER_MIDDLEWARES': {
                'scrapy.contrib.spidermiddleware.httperror.HttpErrorMiddleware': 300,
                'scrapy.contrib.spidermiddleware.offsite.OffsiteMiddleware': 400,
                }
            }

        spider = SpiderParse()
        settings = get_project_settings()
        settings.overrides.update(options)
        crawler = Crawler(settings)
        crawler.signals.connect(reactor.stop, signal=signals.spider_closed)
        crawler.install()
        crawler.configure()
        crawler.crawl(spider)
        crawler.start()
        log.start(logfile="results.log", loglevel=log.DEBUG, crawler=crawler, logstdout=False)
        reactor.run()

Answer the question

In order to leave comments, you need to log in

1 answer(s)
M
Misha Krinkin, 2014-07-02
@JRazor

> Scrapy - import GUI in order to send information to QLabel and second window.
and for this you import your own GUI module, which in turn imports your own Scrapy module, and so on? Why not just connect the necessary part of Qt, create a signal and connect the signal to the GUI already in the GUI module itself?

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question