Crawlers that come to your website have specific identity and this information can be very useful when you are working with log analyzers. You can even study the raw logs and see which crawlers are visiting your website.
Some important crawlers with their specific names:
Google - Googlebot
Google Mediabot - Mediapartners-Google
Google Imagebot - Googlebot-Image
Google Adsbot - Adsbot-Google
Googlebot Mobile - Googlebot-Mobile
Google Search Appliance Crawler: GSA-crawler
Google Feedfetcher: Feedfetcher-Google
Yahoo - Slurp
Msn - Msnbot
Altavista - Scooter, Mercator
Alexa - ia_archiver
Askjeeves - Ask Jeeves
Excite - ArchitextSpider
Looksmart - MantraAgent
One Comment
You made some Good points there. I did a search on the topic and found most people will agree.