I might also add that I did consider that my decision to block Microsoft search bots from my site could negatively impact my site’s traffic. I decided to go ahead with it since Live/Bing/<whatever Microsoft wants to call it today> has such a small portion of the search market, that any impact would be minimal.

http://lessig.org/blog/2008/07/netroots_nation_keynote.html

Someone mailed me about this a few hours ago, saying:

Regarding the part of your article on the denial of service attack by the msnbot against some Perl-related sites, I find it entirely credible. I have a lot of images on my own Website and, several years ago, I noticed that the msnbot would often enough download all of the images with little or no pause. I felt that was exceptionally rude behaviour which no other search engine does. I put it down to Microsoft’s insensitivity and arrogance.

At first, I tried excluding them via robots.txt but the frequent hits from msnbot still seemed to come. Apparently, others have had this problem:

http://www.webmasterworld.com/forum93/411.htm
http://www.webmasterworld.com/msn_microsoft_search/3065473.htm

I then felt that my only recourse was to make a best effort to firewall Microsoft from accessing my site. I first tried to do the blocking narrowly but soon found out that their search engine bots run from many diverse IP ranges. I haven’t had this problem since I adopted a more broad-based block on Microsoft’s IP ranges. In a way, it’s regretable since perhaps some Microsoft employee might want to access my site for the same reason that anyone else would do but, in the end, it’s just karma.

http://cmeerw.org/blog/594.html

Evil M$…