M
M
Max Payne2019-06-04 13:38:08
Python
Max Payne, 2019-06-04 13:38:08

Why does docker-scrapyd use Python 2.7 instead of Python 3.7?

Using what seems to be a nice updatable vimagick/scrapyd image along with thecjw/scrapydweb , the .egg file is built for Python 3.7 and uses type annotations. Error loading .egg in scrapyd

{
    "auth": null,
    "message": "See details below",
    "node_name": "2d354ebaf1d7",
    "status": "error",
    "status_code": 200,
    "url": "http://scrapyd:6800/addversion.json",
    "when": "2019-06-04 13:28:25"
}

Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/local/lib/python2.7/dist-packages/scrapyd/runner.py", line 40, in <module>
    main()
  File "/usr/local/lib/python2.7/dist-packages/scrapyd/runner.py", line 37, in main
    execute()
  File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 149, in execute
    cmd.crawler_process = CrawlerProcess(settings)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 264, in __init__
    super(CrawlerProcess, self).__init__(settings)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 142, in __init__
    self.spider_loader = _get_spider_loader(settings)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/crawler.py", line 351, in _get_spider_loader
    return loader_cls.from_settings(settings.frozencopy())
  File "/usr/local/lib/python2.7/dist-packages/scrapy/spiderloader.py", line 61, in from_settings
    return cls(settings)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/spiderloader.py", line 25, in __init__
    self._load_all_spiders()
  File "/usr/local/lib/python2.7/dist-packages/scrapy/spiderloader.py", line 47, in _load_all_spiders
    for module in walk_modules(name):
  File "/usr/local/lib/python2.7/dist-packages/scrapy/utils/misc.py", line 63, in walk_modules
    mod = import_module(path)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/tmp/JaAuto-2019-06-04T10_26_36-oRHPI3.egg/JaAuto/spiders/__init__.py", line 17
    def parse(self, response: scrapy.http.HtmlResponse):
                            ^
SyntaxError: invalid syntax

Which is logical, because for some reason scrapyd decided to use Python 2.7, however, in the Dockerfile of this image , there is no hint of using Python 2.7, that is, it uses the third version.
The rest of the dockerhub scrapyd images are obviously outdated and have not been updated for several years, how can I fix this error?

Answer the question

In order to leave comments, you need to log in

1 answer(s)
C
chupasaurus, 2019-06-04
@YardalGedal

Hypothesis: because the image is based on Debian and everything is installed via pip instead of pip3.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question