Robots.txt File
The robots.txt file is used to prevent search
engine spiders from indexing either your entire
web site, or specific pages and/or directories.
The following robots.txt file would prevent
your entire web site from being indexed:
User-agent: *
Disallow: /
This would be useful if your site is still
under construction.
You can also use the robots.txt file to
prevent specific spiders from indexing your web
site, by using the spider's name you wish to
exclude, as in the following example:
User-agent: spider-name
Disallow: /
META ROBOTS Tag
Most spiders will honour the META robots tag,
and won't index the page that it appears
on. If you wish to use this tag, simply
copy the following text, and paste it into your
page.
<META NAME="robots"
CONTENT="noindex">
Use this tag in each page that you don't wish
to have indexed by search engines spiders.
Related Pages
