How odd, I was looking at the statistics for this web site and I see that I get absolutely hammered by the search engines:

Top Total User Agents
#HitsUser Agent
14450235.25%Mozilla/5.0 (compatible; Yahoo! Slurp/3.0; http://help.yahoo.com/help/us/ysearch/slurp)
23257125.80%Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
31419011.24%Mozilla/5.0 (Twiceler-0.9 http://www.cuil.com/twiceler/robot.html)
468885.46%Mozilla/4.0 (compatible; Vagabondo/4.0; webcrawler at wise-guys dot nl; http://webagent.wise-guys.nl/; http://www.wise-guys.
514811.17%Yandex/1.01.001 (compatible; Win16; I)
614431.14%Mozilla/5.0 (compatible; DotBot/1.1; http://www.dotnetdotcom.org/, crawler@dotnetdotcom.org)

So, this means that 101075 hits are from search engines out of the total 126256 hits or that search engines make up exactly 80% of the traffic to this site. Nice that Yahoo and Google love me so much but the 1GB of content they downloaded over 1GB of content. Would be nice if they could tone it down a bit. I’ve already adjusted my crawl rate on the engines but I seem to have missed Yahoo and now have made some efforts there to correct that.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes:

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

This site uses Akismet to reduce spam. Learn how your comment data is processed.