U
U
user.2018-11-28 23:53:45
Python
user., 2018-11-28 23:53:45

Python(scrapy) am I doing everything right?

Good evening,
I'm trying to parse this page vulners.com as a parameter I pass CVE (for example: vulners.com/cve/CVE-2017-0147), then on this page I try to get the entire contents of the DIV container "vulners-card-text", with python and scrapy I encountered for the first time, I compiled this script using examples on the network:

#!/usr/bin/python
import scrapy
from scrapy.crawler import CrawlerProcess

"""EX:CVE-2017-0147"""
class PythonEventsSpider(scrapy.Spider):
    name = 'pythoneventsspider'
    start_urls = ['https://vulners.com/cve/'+sys.argv[1], ] 

    events_dict = {}
    def parse(self, response):
        i = 0
        for events in response.xpath('//div[contains(@class, "vulners-item-full")]'):

            event_type = events.xpath('.//div[contains(@class, "vulners-item-description")]').extract_first()
            event_title = events.xpath('.//div[contains(@class, "vulners-item-cpe")]').extract_first()
            event_desc = events.xpath('.//div[contains(@class, "vulners-item-references")]').extract_first()
            events_dict[i] = [event_type, event_title, event_desc]
            i += 1

if __name__ == "__main__":
    process = CrawlerProcess({'LOG_LEVEL': 'ERROR'})
    process.crawl(PythonEventsSpider)
    spider = next(iter(process.crawlers)).spider
    process.start()

But when I run (./pars.py CVE-2017-0147) I see:

Traceback (most recent call last):
File "./pars.py", line 6, in
class PythonEventsSpider(scrapy.Spider):
File "./pars.py", line 8, in PythonEventsSpider
start_urls = [' https:/ /vulners.com/cve/ '+sys.argv[1], ]
NameError: name 'sys' is not defined

Purpose: As a result, I want to get an html file with the contents of the container ...

Answer the question

In order to leave comments, you need to log in

1 answer(s)
V
Vladimir Kuts, 2018-11-29
@nekolov

Add
somewhere at the beginning

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question