Has Google indexed what is hidden in robots.txt?

Question

Search Engine Optimization

Hello everyone, we launched the site, but Google indexed what is hidden in robots.txt. Maybe it's because the sitemap has page data? What to do in this case?

Answer 1

On the page you want to hide, simply put the noindex tag. For some reason, robots rules don't work for everyone.

Answer 2

Do not post pages that you do not want to be available to search engines with available HTTP GET requests, by the way, and even this may not be a guarantee, there were scandals that extensions-panels-bars for the browser, with which at one time everyone was not too lazy to stuff users' browsers, and so for the sake of analytics, they collected all the pages, incl. POST requests

Answer 3

check in the webmaster if the pages are closed from indexing

Answer 4

Google has indexed what is hidden in robots.txt. Maybe it's because the sitemap has page data? What to do in this case?

This is a normal situation for Google PS.
Therefore, if the pages have already been indexed and are shown in Google's search results, placing the robots meta tag (with parameters: <meta name=“robots” content=“noindex,follow”>) before the closing /head on the page will not give anything. The pages will still be in the search engine index.
To correct the situation, you must:

1. remove prohibiting directives in robots.txt for pages that you want to block from search engine indexing;
   
2. set before the closing /head on all pages of the site that you decide to close in the robots: meta tag <meta name=“robots” content=“noindex,follow”>.