What is robots.txt?
robots.txt
is a text file which can be used to restrict web robots to accessing your
web site only in ways of which you approve.
This robots.txt file blocks Google's Imagebot from the entire web site:
User-agent: Googlebot-Image
Disallow: /
For more information on robots.txt, read A
Standard for Robot Exclusion.
Check the Syntax of your robots.txt
Several web-based tools are available which will retrieve the robots.txt
file from your web site and check it for syntax errors.
Bad
Robots
Some web robots will use up considerable amounts of bandwidth and system
resources, while returning little or no practical benefit to the web site
owner.
For your convenience, we maintain a list of some of those bad
robots in robots.txt format.
|
