User-agent: Gigabot Disallow:/ User-agent: Scrubby Disallow:/ # Begin Exclusion From Directories from robots.txt Disallow: /cgi-bin/ Disallow: /BSKT_WWW/ Disallow: /blog/ Disallow: /chat/ Disallow: /css/ Disallow: /images/ Disallow: /wb/ Disallow: /cwbd/ Disallow: /htm-lib/ Disallow: /bsw/ Disallow: /htmlrotate/ Disallow: /CWWW/ Disallow: /FOR_WWW/ Disallow: /FOR_WWW1/ Disallow: /FOR_WWW2/ Disallow: /FOR_WWW1/ Disallow: /HKY_WWW/ Disallow: /includes/ Disallow: /mathscribe/ Disallow: /OT_WWW/ Disallow: /PDFfiles/ Disallow: /rss/ Disallow: /samples/ Disallow: /search/ Disallow: /scgi-bin/ Disallow: /wb/ Disallow: /newdemo/