Generally every website need traffic from search engines, which is the major source of business for any organisation. But there will be few situations where a webmaster would like to restrict few parts / complete site to various search engines.

Webmaster can restrict google search bot, yahoo search bot, msn search bot or any other search bot from indexing the content of the website. This is possible using Robots Exclusion Protocol. Web site owners use the /robots.txt file to give instructions about their site to web robots.

To restrict access to entire site, simple create a txt file with following content :-

User-agent: *
Disallow: /

Save this file as robots.txt and upload to your website using FTP.

Post By Editor (2,827 Posts)

Website: →