-
A text file present in the root directory of a site which is used to
control which pages are indexed by a robot. Only robots which comply with
the Robots Exclusion Standard will follow the instructions contained in this
file.
www.1stsearchranking.net/glossary.htm
-
A file on a web site in the root directory of a website that is used to
control which spiders have access to which pages within a website. When a
spider or robot connects to a website, it checks for the presence of a
robot.txt. Only spiders that adhere to the Robots Exclusion Standard will
obey a robots.txt command file There are several specific fields in a
robots.txt such as User-agent specifies which User Agents are allowed to
access the site and "Allow/Disallow" specifies which directories a spider
may access.
www.azatiko.com/glossary/r.php
-
The file name utilized by the robot exclusion protocol. Web robots
download this file from the server’s document root and parse it for
instructions on what to index and not to index. The case of the file name
does not matter, but it must exist in the document root.
www.rietta.com/robogen/help/glossary.htm
-
A file used to keep web pages from being indexed by search engines.
www.advibemedia.com/html/Search-Engine-Placement/SEO-SERPs.html
-
a file used to exclude some or all robots from crawling some or all the
files or directories on a website. This file should be placed in your
website's root directory.
abelgraphics.co.uk/seo/glossary.php
-
The robots exclusion standard or robots.txt protocol is a convention to
prevent well-behaved web spiders and other web robots from accessing all or
part of a website. The information specifying the parts that should not be
accessed is specified in a file called robots.txt in the top-level directory
of the website.
en.wikipedia.org/wiki/Robots.txt