How to allow access to the robots.txt file only to search robots?

Question

PHP

How to allow access to the robots.txt file only to search robots?

Answer 1

www.useragentstring.com/pages/Crawlerlist I will
add - also do a reverse DNS to check that this is exactly Google

Answer 2

No way. There is no way to know if a user is a robot.

Answer 3

You probably don't want users to know your internal site structure. To remove this problem - transfer everything closed to one of the directories (partitions) and close it entirely. Then you won’t open the structure and explain to the robot that you don’t need to go there.
Defining search bots is as easy as shelling pears - almost all of them indicate their identifiers in the User-Agent, moreover, they go from certain ip subnet ranges. Googling a list of both is pretty easy.
Another question is that if you do not accidentally give access to one of the bots, then it will easily index all your pages and put them in the public domain. So it is better not to use this method.

Answer 4

UserAgentList
Crawlers IP List