How to add Keywords to Robots.txt File – Block url based on keywords in Google Crawl
Its very interesting, You can block certain keywords in robots.txt so that bots will not index such urls which is having that particular keyword on the url. This approach is good for sites like user driven , classifieds . etc.
To Block a keyword just put the following Line in your robot file
Disallow: *xnxxx* Disallow: *tupe8* Disallow: *petardas* Disallow: *nxnn* Disallow: *kamapischi*
In red i specified the keywords.
Note: You have to be vary careful while playing with your robot.txt file
Latest posts by Gishore James (see all)
- Microsoft Office for iPhone – Changing Strategy of Microsoft - June 16, 2013
- Guidelines for International websites from Google - June 13, 2013
- Google is to Change the rankings of smartphone search results - June 12, 2013