Where do search robots get URLs to crawl?

Question

Search engines

I have no idea where search engines get page addresses for parsing and indexing? Do they brute force, or is there a certain place where all the addresses of pages published on the web are located? Can I find out all available pages for a specific domain?

Answer 1

users themselves upload a list of pages on their site for initial indexing.
plus, the robot periodically parses links from pages that it already has in the index, and then runs through them, etc.

Answer 2

I will add that in some cms (Wordpress) there is a built-in ping that reports to the update services when creating and updating pages. Link .

Answer 3

Can I find out all available pages for a specific domain?

This is called a sitemap, for example:
https://qna.habr.com/sm-questions.xml