How to add Keywords to Robots.txt File – Block url based on keywords in Google Crawl

Its very interesting, You can block certain keywords in robots.txt so that bots will not index such urls which is having that particular keyword on the url. This approach is good for sites like user driven , classifieds . etc.

To Block a keyword just put the following Line in your robot file

Disallow: *xnxxx*
Disallow: *tupe8*
Disallow: *petardas*
Disallow: *nxnn*
Disallow: *kamapischi*

In red i specified the keywords.

Note: You have to be vary careful while playing with your robot.txt file

Bio
Latest Posts

Gishore James

Manager Technology at Quikr

I am a highly successful, competent team member with a background of extensive hands-on project implementation experience. My success has been both internal to the companies to which I’ve contributed and external to clients and customers.Having the knowledge of setting up efficient & cost effective technology solutions, internet innovations, large scale IT implementations, helps me to find out the unique one in the way of my career.