S
S
Sergey Bard2017-09-13 22:12:08
Python
Sergey Bard, 2017-09-13 22:12:08

How to write data to MySQL BD in scrapy?

Hello. Please help me write the data to the database, I just started working with scrapy.
I scan all links from the sitemap, like this

from scrapy.spiders import SitemapSpider
class MySpider(SitemapSpider):
  name = 'jobs'
  sitemap_urls = ['http://example.com/sitemap.xml']
  item = []
  def parse(self, response):
    item["title"] = response.css("title ::text").extract_first()
    item["description"] = response.css("meta[name=\"description\"] ::attr(content)").extract_first()
    item["url"] = response.url
    item["status"] = response.status
    return item

Added these lines to settings.ts
ITEM_PIPELINES = {
    'craigslist.pipelines.CraigslistPipeline': 300,
}
DB_SETTINGS = {
  'db': 'db',
  'user': 'user',
  'passwd': 'passwd',
  'host': 'host',
}

in pipelines.py such code
class CraigslistPipeline(object):
  def process_item(self, item, spider):
    query = ("INSERT INTO table1 (title, description, code, url) VALUES (%s, %s, %s, %s)")
    self.cursor.execute(query,  (
                    item["title"],
                    item["description"],
                    item["code"],
                    item["url"],
                  )
                        )
    self.conn.commit()
    return item

and in middlewares.py like
1VVJ1k.jpg
this, when I do this, it shows me something like this
NameError: name 'item' is not defined
, as I understand it, the error is in class MySpider, I searched for examples on the net, but could not find them everywhere show connection examples, etc., but no one shows how to transfer data for writing from the parser class to the intermediary.
Please help me which module I need to connect or fix if I wrote wrong somewhere.

Answer the question

In order to leave comments, you need to log in

1 answer(s)
A
Astrohas, 2017-09-13
@serg_small_developer

Add self before item in parse method

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question