Answer the question
In order to leave comments, you need to log in
Does the search robot download music files?
Good afternoon!
Imagine that there is a site that stores links to music files.
A search robot gets to the site and sees all these links. Will he cross them?
If so, will it download all files, or will it see that the title is equal to audio/mpeg, where will it stop downloading?
As I understand it, does it make sense to disable a certain folder with files in robots.txt so that the spider does not climb into them?
Answer the question
In order to leave comments, you need to log in
Didn't find what you were looking for?
Ask your questionAsk a Question
731 491 924 answers to any question