User-agent: * #block directories from spiders Disallow: /t/js Disallow: /cgi-bin Disallow: /doc Disallow: /mp3 Disallow: /images