Do It Yourself SEO

  1. What is robots.txt file?
    It is a file located in your website's root directory that indicates which parts of your website are not allowed to be crawled and indexed by the Search Engines Crawler. This is just to make the crawler not crawled your web server. Please note that robots.txt is NOT intended to HIDE your web pages from the search engine results. Because other pages may point to your pages, thus your page will be shown in the search engine results. You should password-protect your directories that you don't want people to access so that it won't be found in the search engine results (or if it is found, people still cannot access it because of the password).
  2. Filename MUST be: robots.txt
    NOT: Robots.txt nor ROBOTS.TXT nor robots.TXT nor ROBOTS.txt
  3. File (robots.txt) MUST be saved as a TEXT file.
  4. Example of robots.txt
    Create ref="seo-learning-robots-txt.html">robots.txt file with the following code inside the file:

    User-agent: *
    Disallow: /cgi-bin/                     # disallow this directory
    Disallow: /tmp/                          # disallow this directory

    User-agent: googlebot              # all Google services
    Disallow: /private/                     # disallow this directory

    User-agent: googlebot-news     # only the news service
    Disallow: /                                 # disallow everything
  5. Test your robots.txt file with Google's robots.txt Tester.
    Here is the Guideline to text your robots.txt file from Google.
  6. Submit your new robots.txt file to Google
    The purpose is to ask Google more quickly crawl and index your robots.txt file.
  7. References:



This page is always updated with new information. Please bookmark it to go back for newer and fresh information about Search Engine Optimization..

Go back to SEO Learning Center.


Home | Submit a Site | SEO Tools | Do It Yourself SEO | Internet Marketing

Copyright All Rights Reserved.
eSitus Media Group