“webinar”/ “VOLUUM”/
Welcome to Our Community
Wanting to join the rest of our members? Feel free to sign up today.

Spiders List

Discussion in 'Search Engine Optimisation' started by ovi, Sep 12, 2004.

  1. ovi

    ovi Guest

    Maybe someone will find interesting and util this list:

    Alexa Research - alexa.com - ia_archiver
    AltaVista - altavista.com - Scooter/3.3
    AltaVista - altavista.com - Scooter/3.3.vscooter
    Ask Jeeves - askjeeves.com - Mozilla/2.0 (compatible; Ask Jeeves)
    Ask Jeeves/Teoma - teoma.com - Mozilla/2.0 (compatible; Ask Jeeves/Teoma)
    FAST - alltheweb.com - FAST-WebCrawler/3.8
    FAST - alltheweb.com - FAST-WebCrawler/3.x Multimedia
    FAST/Overture - alltheweb.com - FAST-WebCrawler/3.8
    gigablast.com - gigablast.com - Gigabot/1.0
    Google - google.com - Googlebot/2.1
    Google - google.com - Googlebot-Image/1.0
    Google - google.com - Mediapartners-Google/2.1
    Google - google.com - compatible; Googlebot/2.1
    Google - google.com - Googlebot/2.1
    Infoseek - infoseek.com - SideWinder/2.0B
    Inktomi a Yahoo Company - inktomi.com - Slurp/si
    Looksmart - looksmart.com - ZyBorg/1.0 Dead Link Checker
    Looksmart - looksmart.com - ZyBorg/1.0
    LookSmart, Ltd - looksmart.com - grub-client-
    Microsoft Corporation - msn.com - msnbot/0.11
    Microsoft Corporation - msn.com - MSRBOT/0.1
    NameProtect Inc. - nameprotect.com - NPBot
    NHN Corp - naver.com - Cowbot-0.1.1
    NHN Corp - naver.com - NaverBot-1.0
    Openfind Taiwan - openfind.com.tw - Openbot/3.0+
    walhello.com - walhello.com - appie 1.1
    www.entireweb.com - entireweb.com - Speedy Spider
    www.girafa.com - girafa.com - Girafabot
    Yahoo! - yahoo.com - Yahoo! Slurp
    Yahoo/Overture - yahoo.com - Yahoo-MMCrawler/3.x

    If you know more, please add :)

    Ovi
     
  2. Voluum
  3. Paul_KY

    Paul_KY Affiliate affiliate

    1,399
    0
    0
    My advice, know which bots to block.

    I love SE bots, bet they're not ALL GOOD.

    Trust me on this one, please.

    Thank You

    Hope everyone had a great weekend.

    Paul
     
  4. ovi

    ovi Guest

    Right

    Right, I agree with you, not all the spiders are ok :) Specialy the spam spiders :(

    Ovi
     
  5. Paul_KY

    Paul_KY Affiliate affiliate

    1,399
    0
    0
    Yeah :roll:

    And what about those no-name SE bots crawling your site to death, draining your bandwidth :roll:

    Paul
     
  6. baggiho

    baggiho Affiliate affiliate

    4
    0
    0
    how can make the SE to spider my website?
     
MI