S
S
Sergey2020-09-01 19:20:27
Python
Sergey, 2020-09-01 19:20:27

What is the difference between Scrapy and BeautifulSoup+Requests?

I have little experience in using both options, but do not really understand the difference between them.
Which option is better to use and why?

Answer the question

In order to leave comments, you need to log in

1 answer(s)
I
Ivan Yakushenko, 2020-09-01
@impakashi

BeautifulSoup is just an html parser.
Requests is just a library for making HTTP requests.
Scrapy is a scraping/crawling framework that out of the box can do asynchrony, page traversal, data presentation and much more, and it does it quickly and efficiently.
So, there is something - figured it out.
If you need to parse something once in a small amount - requests + lxml (yes, it's better than bs at least in what it can do in xpath) more than.
Large volume? aiohttp + lxml (but here you already need to know at least a little in python).
Need to build a crawler (or many crawlers) that will run daily and go through a bunch of pages? Cosplaying a bicycle engineer is great, but scrapy is better.

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question