Spiders List

[ovi]

Maybe someone will find interesting and util this list:

Alexa Research - alexa.com - ia_archiver
AltaVista - altavista.com - Scooter/3.3
AltaVista - altavista.com - Scooter/3.3.vscooter
Ask Jeeves - askjeeves.com - Mozilla/2.0 (compatible; Ask Jeeves)
Ask Jeeves/Teoma - teoma.com - Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
FAST - alltheweb.com - FAST-WebCrawler/3.8
FAST - alltheweb.com - FAST-WebCrawler/3.x Multimedia
FAST/Overture - alltheweb.com - FAST-WebCrawler/3.8
gigablast.com - gigablast.com - Gigabot/1.0
Google - google.com - Googlebot/2.1
Google - google.com - Googlebot-Image/1.0
Google - google.com - Mediapartners-Google/2.1
Google - google.com - compatible; Googlebot/2.1
Google - google.com - Googlebot/2.1
Infoseek - infoseek.com - SideWinder/2.0B
Inktomi a Yahoo Company - inktomi.com - Slurp/si
Looksmart - looksmart.com - ZyBorg/1.0 Dead Link Checker
Looksmart - looksmart.com - ZyBorg/1.0
LookSmart, Ltd - looksmart.com - grub-client-
Microsoft Corporation - msn.com - msnbot/0.11
Microsoft Corporation - msn.com - MSRBOT/0.1
NameProtect Inc. - nameprotect.com - NPBot
NHN Corp - naver.com - Cowbot-0.1.1
NHN Corp - naver.com - NaverBot-1.0
Openfind Taiwan - openfind.com.tw - Openbot/3.0+
walhello.com - walhello.com - appie 1.1
www.entireweb.com - entireweb.com - Speedy Spider
www.girafa.com - girafa.com - Girafabot
Yahoo! - yahoo.com - Yahoo! Slurp
Yahoo/Overture - yahoo.com - Yahoo-MMCrawler/3.x

If you know more, please add

Ovi

 

[Paul_KY]

My advice, know which bots to block.

I love SE bots, bet they're not ALL GOOD.

Trust me on this one, please.

Thank You

Hope everyone had a great weekend.

Paul

 

[ovi]

Right

Right, I agree with you, not all the spiders are ok Specialy the spam spiders

Ovi

 

[Paul_KY]

Yeah :roll:

And what about those no-name SE bots crawling your site to death, draining your bandwidth :roll:

Paul

 

[baggiho]

how can make the SE to spider my website?