User-agent: * Disallow: /.smileys/ Disallow: /abicons/ Disallow: /abmasterd/ Disallow: /bottom.html Disallow: /cgi/ Disallow: /cgi-bin/ Disallow: /cgi-sys/ Disallow: /chat/ Disallow: /counter/ Disallow: /counts/ Disallow: /General/ Disallow: /groovy/ Disallow: /guestbook/ Disallow: /guestbook.php Disallow: /illusion/ Disallow: /incoming/ Disallow: /logs/ Disallow: /tarfiles/ Disallow: /test/ Disallow: /uuforum/ Disallow: /zips/ #we receive no hits from these sources User-agent: speedy Disallow: / User-agent: MLBot Disallow: / User-agent: ScoutJet Disallow: / User-agent: envolk Disallow: / User-agent: Exabot Disallow: / User-agent: Sosospider Disallow: / User-agent: nodestackbot Disallow: / User-agent: FreeFind Disallow: / User-agent: turnitinbot Disallow: / User-agent: Yandex Disallow: / User-Agent: YodaoBot Disallow: / User-agent: twiceler Disallow: / User-agent: discobot Disallow: / User-agent: VoilaBot Disallow: / User-agent: spbot Disallow: / # the following is borrowed from the robots.txt for http://www.wikipedia.org/ User-agent: Mediapartners-Google* Disallow: / User-agent: UbiCrawler Disallow: / User-agent: DOC Disallow: / User-agent: Zao Disallow: / User-agent: sitecheck.internetseer.com Disallow: / User-agent: Zealbot Disallow: / User-agent: MSIECrawler Disallow: / User-agent: SiteSnagger Disallow: / User-agent: WebStripper Disallow: / User-agent: WebCopier Disallow: / User-agent: Fetch Disallow: / User-agent: Offline Explorer Disallow: / User-agent: Teleport Disallow: / User-agent: TeleportPro Disallow: / User-agent: WebZIP Disallow: / User-agent: linko Disallow: / User-agent: HTTrack Disallow: / User-agent: Microsoft.URL.Control Disallow: / User-agent: Xenu Disallow: / User-agent: larbin Disallow: / User-agent: libwww Disallow: / User-agent: ZyBORG Disallow: / User-agent: Download Ninja Disallow: / User-agent: wget Disallow: / User-agent: grub-client Disallow: / User-agent: k2spider Disallow: / User-agent: NPBot Disallow: / User-agent: WebReaper Disallow: /