A
A
Andrey Pavlenko2016-07-05 13:23:50
Google
Andrey Pavlenko, 2016-07-05 13:23:50

Does the search robot download music files?

Good afternoon!
Imagine that there is a site that stores links to music files.
A search robot gets to the site and sees all these links. Will he cross them?
If so, will it download all files, or will it see that the title is equal to audio/mpeg, where will it stop downloading?
As I understand it, does it make sense to disable a certain folder with files in robots.txt so that the spider does not climb into them?

Answer the question

In order to leave comments, you need to log in

Didn't find what you were looking for?

Ask your question

Ask a Question

731 491 924 answers to any question