Some times if you do customized crawl rate in webmaster tools as high , your site may experience very high load of Traffic. And also you can see that some of the httpd threads are taking 90% Â of CPUÂ utilization. This happens becauseÂ GoogleÂ is trying to crawl more pages from your site. It may also lead to slow browsing orÂ unavailabilityÂ of your site to the end users. What is the solution for this kind of problem?
Yes there are some kind of solution to it.
- Set the crawl rate to default (Determine byÂ GoogleÂ ) This is the best solution to reduce theÂ loadÂ on the server. Under this settings,Â GoogleÂ will calculate your site response and determine the crawling speed.
- Dedicate someÂ web serversÂ Only for Search Engines.In thisÂ scenario you need more servers only dedicated for Search engines. This effort require time as well as money. Under this Â you have toÂ parallelÂ set upÂ Â some servers only for google, you need a Load balancer to segregate the traffic between normall users as well as Serch engines.
The second method is very good for companies who are mainly focusing on SEO.