A
A
Andrey Kovalchuk2017-02-05 17:27:35
PostgreSQL
Andrey Kovalchuk, 2017-02-05 17:27:35

Scrapy + django-item + postgresql (sqlalchemy), how to map a class?

I use this stack in a project. The following error is thrown:


sqlalchemy.orm.exc.UnmappedInstanceError: Class news_scrap.items.NewsItem' is not mapped.

It seems that I understand what's wrong, but how to fix it, I'll never know. The Internet tells you how to work if you do not have django-item, and with it - xs. I hope for you. Below are the scrapy files.
Items.py
from scrapy_djangoitem import DjangoItem
from main.models import News

class NewsItem(DjangoItem):
    django_model = News
    __tablename__ = "main_news"

Models.py
from sqlalchemy import *
from sqlalchemy.engine.url import URL
from . import settings

def db_connect():
    """
    Performs database connection using database settings from settings.py.
    Returns sqlalchemy engine instance
    """
    return create_engine(URL(**settings.DATABASE))

Pipelines.py
from sqlalchemy.orm import sessionmaker
from .items import NewsItem
from .models import db_connect

class NewsScrapPipeline(object):
    def __init__(self):
        """
        Initializes database connection and sessionmaker.
        Creates deals table.
        """
        #log.INFO('First Point')
        engine = db_connect()
        self.Session = sessionmaker(bind=engine)

    def process_item(self, item, spider):

        session = self.Session()
        news = NewsItem(**item)
        #log.INFO('This Point')

        try:
            session.add(news)
            session.commit()
        except:
            session.rollback()
            raise
        finally:
            session.close()

        return item

If you need any more plots when, say.

Answer the question

In order to leave comments, you need to log in

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question