It is very important to test your robots.txt file before uploading to your website root directory. The robots file contains the instructions to the crawling bots regarding what to crawl and index in the website. If you have any misconfiguration then your site will not get indexed in the search engine.

Here we are going to test your robots file in the Google Webmaster Tool center.

First you need to loging to google webmaster center with your user id and password.  Once login,  you just click on your profile and expand site configuration tab. Then click on Crawler Access.You will get a window like below.

w1 Here in the above picture you can see the contents of your robots.txt file. please copy paste your changed robots.txt file here. Then please write your website name in the url field and select the user agent shown as below screenshot. ( Try to test for all the agents otherwise some may be blocked for adsense agents, so the add will not appear in that particular page.)

w2Once you have given the url and selected the agent click on TEST button, you will get a test result as shown below.

w3In the above result you can see that my url ( is allowed. Now you check allmost all paterns of your website url like view page, search page, etc.

A Video on Creating Your Robots.txt File

A video on Uploading Robots.txt File

Post By Gishore J Kallarackal (37 Posts)

Lead Datacenter Operations at, Having the knowledge of setting up efficient & cost effective technology solutions, internet innovations, large scale IT implementations, helps me to find out the unique one in the way of my career.

Website: →