A
A
Artyom Innokentiev2016-03-03 13:46:25
Django
Artyom Innokentiev, 2016-03-03 13:46:25

Django + Scrapy: which approach to take?

Task: it is necessary to save data from Scrapy parsed data through Django ORM.
Searching, I found several approaches:
1) Django Item
2) django-dynamic-scraper
3) Initialize Django from under Scrapy settings
Which approach should I choose? Anyone have experience with either of these approaches (pros or cons)?

Answer the question

In order to leave comments, you need to log in

2 answer(s)
S
sim3x, 2016-03-03
@artinnok

Try everything
With low RPS and good data validation, you can immediately put into djanga

D
Dimonchik, 2016-03-03
@dimonchik2013

The problem with Scrapy data is that it can be incomplete, hence - only NoSQL DB during initial saving, the data then needs to be processed one way or another - get rid of duplicates, empty ones, determine the required, but missing ones,
I would not contact such a combine, scrapy separately - django processing and presentation - separately

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question