Some times if you do customized crawl rate in webmaster tools as high , your site may experience very high load of Traffic. And also you can see that some of the httpd threads are taking 90%  of CPU utilization. This happens because Google is trying to crawl more pages from your site. It may also lead to slow browsing or unavailability of your site to the end users. What is the solution for this kind of problem?

Yes there are some kind of solution to it.

  • Set the crawl rate to default (Determine by Google ) This is the best solution to reduce the load on the server. Under this settings, Google will calculate your site response and determine the crawling speed.
  • Dedicate some web servers Only for Search Engines.In this scenario you need more servers only dedicated for Search engines. This effort require time as well as money. Under this  you have to parallel set up  some servers only for google, you need a Load balancer to segregate the traffic between normall users as well as Serch engines.

The second method is very good for companies who are mainly focusing on SEO.

Post By Editor (2,827 Posts)

Website: →