What to study for web crawling?

Question

Python

Good afternoon!
What to study for web crawling?
As I understand it, JP is python. But what technologies / libraries to study, and whether there is any specific route for studying this direction, is not clear.
Thanks in advance!
PS: who worked / is working with this direction - are there any prospects?

Answer 1

https://lxml.de/
https://scrapy.org/
https://www.crummy.com/software/BeautifulSoup/
That's enough for a start, there are a lot of examples of using these libraries on the net.
+ articles on Habré optional:
https://habr.com/en/company/ods/blog/346632/
Literature:
Website scraping with Python, Ryan Mitchell.
Also, sometimes Phantomjs or Selenium are needed.

Answer 2

If you are going to data science / data mining, then there are prospects
to study:
It is also not superfluous to know SQL, to know how to "prepare" data