Blocking SearchBots | robots.txt

robots.txt and how to use it.
At some point in time you may want to block searchbots from certain pages you have at your site. You may want to block searchbots from pages that are for internal use only as well as directories that you store scripts at as well as pages that are still under construction.

To block searchbots you will need to set up a robots.txt at your site. The searchbots will look for the robots.txt file in your root domain ie.. http://www.yoursitedomain.com/robots.txt so be sure to add one if you have not done so.

The format for the robots.txt file consists of records. Each record consists of two fields : a User-agent line and one or more Disallow: lines.

The following allows all robots to visit all files because the wildcard "*" specifies all robots.

User-agent: *

Disallow:

This one keeps all robots out.

User-agent: *

Disallow: /

The next one bars all robots from the cgi-bin and images directories:

User-agent: *

Disallow: /cgi-bin/

Disallow: /images/

This one bans whateverbot from all files on the server:

User-agent: whateverbot
Disallow: /

This one bans keeps googlebot from getting at the rayne.html file:

User-agent: googlebot

Disallow: rayne.html

Return to SEO Blog