A
A
Arch-Linux2015-07-23 15:54:28
Search Engine Optimization
Arch-Linux, 2015-07-23 15:54:28

How to set up a robots.txt file for a php site?

Hello, I have been solving this problem with robots.txt for more than a week now, then the problem is solved then it appears, and sometimes it is not solved at all User-agent: *
Disallow: /css
Disallow: /js
Disallow: /42813461165351.php
Disallow: /registrazione .html
Disallow: /igea
Disallow: /04
Allow: /en
Allow: /en
Allow: /es
Allow: /de
Allow: /fr
Allow: /it
Allow: /jp
Allow: /cn
User-agent: Yandex
Disallow: / css
Disallow: /js
Disallow: /42813461165351.php
Disallow: /registrazione.html
Disallow: /igea
Disallow: /04
Allow: /ru
Allow: /en
Allow: /es
Allow: /de
Allow: /fr
Allow: /it
Allow: /jp
Allow: /cn
Host: jeunesse-global.club
User-agent : Googlebot
Allow: /en
Allow: /en
Allow: /es
Allow: /de
Allow: /fr
Allow: /it
Allow: /jp
Allow: /cn I have already done everything I could to solve problems, but it still can’t be solved ------ but one person needs all sites to be displayed in all languages ​​in the search engine, including the main link, I start with / -of course I won’t take a screenshot right away, I’ll throw off the link site: www.jeunesse-global.club, right now at this moment I solved the problem a little, so when I removed the code in the en.php file that changes the language in the browser, then after that the link restriction was removed, in the index.php file, and I think that the restrictions appear due to one code that redirects to another file, right now there is one code that is written in the main file / but I don’t know exactly what kind of file and where it is, but here is the code for this file found by www.google.com/webmasters/joxi.ru/nAyXq46uLWlY2Z.png , according to google law, there should not be links in the search engine that redirect to other pages, so if I put the standard code in robots.txt User-agent: *
Disallow: /css
Disallow: /font
Disallow: /js
User-agent: Yandex
Disallow : /css
Disallow: /font
Disallow: /js
Host: jeunesse-global.club then after that when all links and even non-working links will be indexed, and as far as I remember, only after this code some links began to fall into the restriction this is the first time I created a robots.txt file and wrote this code --
joxi.ru/82Q7Y64ikZalrd.png joxi.ru/L21j0PJIQW3nAX.png I've already tried everything and did it this way and did it differently, and it still turns out like this screenshot
joxi.ru/1A59zPJC3E0MAE.png

Answer the question

In order to leave comments, you need to log in

2 answer(s)
A
Anton, 2015-07-24
@aksm

What is your problem in the end?

A
Anton Shamanov, 2015-07-23
@SilenceOfWinter

robotstxt.org.ru everything you wanted to know but were afraid to ask your mom

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question