How does a search engine work with large amounts of data?

Question

Search engines

Do bots, when crawling web documents, have no scheme for building a search index, such as dividing into categories "animals", "computers", "money", etc.? so that when entering the query "animals" we searched only in the "animals" category, without raising the entire search base, going through each page. Otherwise, this is how many iterations will have to be done to answer the user.

Answer 1

As a first approximation https://yandex.ru/company/technologies/matrixnet/
Well, in general, the entire section
https://yandex.ru/company/technologies
But you will not find any specifics anywhere.

Answer 2

Search engines use at least reverse index technology. I advise you to read

Answer 3

Naive questions for those who have not encountered implementations. Okay, let's start implementing: oops, where do we get all these categories and how to fill them in, okay, somehow they did it all, oops, and then homonyms also surfaced, okay, somehow they all did it. And so, at each implementation iteration, additional complexity appears.