How to make search robots access only what they need in robots.txt?

Question

robots.txt

I read a lot of articles on writing a robots.txt file, but did not understand it. It would be easier for me to ban everything except for the pages that I indicate. What is the best way to do this?

Answer 1

What specifically do you not understand? Everywhere it says the same thing:
disallow

The disallow directive defines specific paths that should be inaccessible to the specified crawlers. If no path is specified, it is ignored.

Usage:
disallow: [path]
allow

The allow directive defines the paths that should be available to the specified crawlers. If no path is specified, it is ignored.

Usage:
allow: [path]

disable EVERYTHING except those pages that I specify

:

allow: /url-1.html
allow: /url-2.html
allow: /2020-08-28/*
disallow: /

Here, for example, we indicate the full path to the page in allow, so it will be allowed,
allow: /2020-08-28/* - all pages in this folder will be allowed for indexing,
disallow: / - Deny ALL