Robots.txt Generator


Default - All Robots are:  
    
Crawl-Delay:
    
Sitemap: (leave blank if you don't have) 
     
Search Robots: Google
  Google Image
  Google Mobile
  MSN Search
  Yahoo
  Yahoo MM
  Yahoo Blogs
  Ask/Teoma
  GigaBlast
  DMOZ Checker
  Nutch
  Alexa/Wayback
  Baidu
  Naver
  MSN PicSearch
   
Restricted Directories: The path is relative to root and must contain a trailing slash "/"
 
 
 
 
 
 
   



Now, Create 'robots.txt' file at your root directory. Copy above text and paste into the text file.


About Robots.txt Generator

Robots.txt Generator Tool:

A tool that would certainly prove extremely helpful to website owners in making their websites Google bot friendly. This sophisticated SEO tool comprised off a user-friendly interface, and you have the choice to choose which things should be covered within the robots.txt file and which is not.

Robots.txt Generator produces a file that is fundamentaly opposite of the sitemap which stipulates the pages to be covered. Hence, the robots.txt syntax is of utmost significance for any site. Each time a search engine crawls a site, it searches for the robots.txt file first that is positioned at the domain root level. Once identified, the crawler will read the file, and afterward identify the directories and files that may be blocked.

Robots.txt - extremely valuable tool that has made some webmaster’s lives easier by supporting them making their sites Google bot friendly.

Using our tool, you can generate a robots.txt file for your site through these few simple and easy steps:

  • By default, all Google robots.txt generator tools are allowed to access your website’s files; you can select the robots you want to refuse or allow the access
  • Select crawl-delay which instructs how much delay must be there in the crawls, allowing you to select your desired delay duration from 5 to 100 seconds. By default, it is set to “No Delay”.
  • If a sitemap already exists for your website, you can paste it in the text field. Instead, you can leave it blank in case you don’t have.
  • A list of search robots is provided, you can choose the ones you wish to crawls your website, or you can say no to the robots you don’t want to crawl your files.
  • The final step is to confine directories. The path should comprise off a trailing slash "/", as the path is comparative to root.

The robots.txt file is beneficial for keeping your search engine spiders from accessing parts files and folders in your website hosting directory that are completely unrelated to your real website content.