Answer the question
In order to leave comments, you need to log in
Error in Google Webmaster. Why is access to the URL blocked in the robots.txt file?
Hello. Faced such a problem. Created in CMS 1C-Bitrix, using built-in services, robot.txt and sitemap.xml files (sitemap), added robot.txt file to Google Webmasters, no errors or warnings were found by Google. By the way, this is how my robot.txt looks like:
User-Agent: *
Disallow: */index.php
Disallow: /bitrix/
Disallow: /*show_include_exec_time=
Disallow: /*show_page_exec_time=
Disallow: /*show_sql_stat=
Disallow: /*bitrix_include_areas =
Disallow: /*clear_cache=
Disallow: /*clear_cache_session=
Disallow: /*ADD_TO_COMPARE_LIST
Disallow: /*ORDER_BY
Disallow: /*PAGEN
Disallow: /*?print=
Disallow: /*&print=
Disallow: /*print_course=
Disallow: /*?action=
Disallow: /*&action=
Disallow: /*register=
Disallow: /*forgot_password=
Disallow: /*change_password=
Disallow: /*login=
Disallow: /*logout=
Disallow : /*auth=
Disallow: /*backurl=
Disallow: /*back_url=
Disallow: /*BACKURL=
Disallow: /*BACK_URL=
Disallow: /*back_url_admin=
Disallow: /*?utm_source=
Host: mysite.by
Sitemap: http ://mysite.by/sitemap.xml
The real fun started after I added the sitemap.xml to Google Webmasters. It turns out that robot.txt blocks absolutely all links that sitemap.xml contains, while at the same time I didn’t pervert over robot.txt, with the help of which manuals I wouldn’t create it again, by the way, the same applies to sitemap.xml, I do it too I tried to create not only using the built-in service, anyway robot.txt blocks all links in sitemap.xml . Has anyone experienced this and knows what needs to be done?
Answer the question
In order to leave comments, you need to log in
Explain what does " robot.txt blocks all links in sitemap.xml " mean?
In your example, there are no rules that would exclude all pages. Perhaps in your sitemap all sections have the /index.php ending , which will be excluded by the first rule.
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question