Parser of all site pages and links in php/python?

Question

Parsing

The guys are faced with the task of collecting all the pages of the site and from all the pages links to files (jpg, pdf, js) and so on.
Any of the two languages ​​php/python.
I searched on github, there are a lot of solutions, but they all have to be redone for themselves.
And I'm more interested in the question of whether there are ready-made solutions so as not to reinvent the wheel.
The easiest option that I found is to take the sitemap generator and remake it for myself. But I'm sure there are already ready-made options, I just could not find them.

Answer 1

scrapy
couldn't think of anything better