H
H
HeadTek2015-07-31 10:17:38
Googlebot
HeadTek, 2015-07-31 10:17:38

How to solve the problem with the inaccessibility of AWS by Googlebot?

Hello!
When checking with Google Webmaster Tools, the service says that the CDN is not available. That is, the *.cloudfront.net address is not available. Accordingly, Googlebot sees the page without CSS and scripts.
AWS removed all entries containing the "?" sign, empty lines from robots.txt, as there is information that the Amazon parser does not accept them.
At the moment, robots.txt on cloudfront.net looks like this:

User-agent: *
Disallow: /*supported_formats.htm
Disallow: /news/*
Disallow: /news_ru/*
Host:  hostname.cloudfront.net

It makes sense to try to remove entries with "?" from robots.txt of our server?
robots.txt of our server:
User-agent: Mediapartners-Google
Disallow: /buy-
Disallow: /download-
Disallow: /*download.html
Disallow: /thank-you-for-installing.html
Disallow: /thanks-for-your-interest.html
User-agent: AdsBot-Google
Disallow: /buy-
Disallow: /download-
Disallow: /*download.html
Disallow: /thank-you-for-installing.html
Disallow: /thanks-for-your-interest.html
User-agent: *
Disallow: /buy-
Disallow: /download-
Disallow: /*formats.html
Disallow: /*requirements.html
Disallow: /*download.html
Disallow: /thank-you-for-installing.html
Disallow: /thanks-for-your-interest.html
Disallow: /*?*
Disallow: /*?
Disallow: /adw/*
Sitemap: http://www.hostname.pl/sitemap.xml
Host: www.hostname.pl
User-agent: Googlebot-Image
Disallow: /

Answer the question

In order to leave comments, you need to log in

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question