Its very interesting, You can block certain keywords in robots.txt so that bots will not index such urls which is having that particular keyword on the url. This approach is good for sites like user driven , classifieds . etc.

To Block a keyword just put the following Line in your robot file

Disallow: *xnxxx*
Disallow: *tupe8*
Disallow: *petardas*
Disallow: *nxnn*
Disallow: *kamapischi*
In red i specified the keywords.
Note: You have to be vary careful while playing with your robot.txt file

Post By Gishore J Kallarackal (2,121 Posts)

Gishore J Kallarackal is the founder of techgurulive. The purpose of this site is to share information about free resources that techies can use for reference. You can follow me on the social web, subscribe to the RSS Feed or sign up for the email newsletter for your daily dose of tech tips & tutorials. You can content me via @twitter or e-mail.

Website: → Techgurulive