Confirm the authenticity of the MSN-bot

Johannes Beus
After Google has already done so, now, MSN has also implemented a method of confirming the authenticity of the MSN-bot. A reason for this is probably the fact that, by now, many scraperbots are using the useragents of the large searchengines to keep from being noticed. Just as with Google, the confirmation runs through a DNS- and Reverse-DNS-lookup. A typical MSN-bot access would look like this:

64.4.8.131 - - [30/Nov/2006:07:19:03 +0100] "GET /robots.txt HTTP/1.0" 200 76 "-" "msnbot/1.0 (+http://search.msn.com/msnbot.htm)"

Resolving the IP we get

hades:~# host 64.4.8.131
Name: livebot-64-4-8-131.search.live.com
Address: 64.4.8.131

Now all that is left to do is check if the host is also pointing to the IP and the authenticity is confirmed:

hades:~# host livebot-64-4-8-131.search.live.com
livebot-64-4-8-131.search.live.com A 64.4.8.131

That fits and the authenticity of this access is granted.

This system, like Google's, is based on the fact that everyone is free to enter whatever they want into the Reverse-DNS-entry – the Zonefiles, which resolve from Host to IP, can only be accessed by the owner of the domain.

For those who do not want to do all of this by hand, you can use this quickly constructed tool:
Confirm the authenticity of a Botaccess. Please note thought that not all reverse-entries are set by Microsoft which will cause some false alarms in the beginning.
Johannes Beus - on Thu (11/30/2006) at 08:50 AM

Add Comment

more
This posting is older than 30 days and therefore closed for new comments.