How to parse all pages of a site?

Question

Python

Let's say I have a site site.site and it has pages site.site/siter, site.site/1, site.site/simn.
How can I sort through them, having only the main site.site link, since there are a lot of pages in my task, there is no way to manually, and the difference in page addresses is similar to the one in the example above.

Answer 1

Let's say you don't know anything about parsing and programming. You have a site.site how do you plan to know that it has a site.site/siter page?
The simplest is a recursive url search on pages with a nesting limit. Open site.site, find everything there, href="([^"]+)"then open all found and search there. Not efficient, but it works.

Answer 2

as a rule, if the site monitors its SEO, then it will have site.site/sitemap.xml or sitemap.html in the public domain.
The name can be different, it can also be specified in the robots.txt file.
Find this file, parse it into links and here you have all the links from the site.