#################################################################################################################### #################################################################################################################### # robots.txt : Documentation by example # # User-agent: webcrawler # Disallow: # # User-agent: lycra # Disallow: / # # User-agent: * # Disallow: /tmp # Disallow: /logs # Disallow: /foo.html # # # The first two lines, starting with '#', specify a comment # # The first paragraph specifies that the robot called 'webcrawler' has nothing disallowed: it may go anywhere. # # The second paragraph indicates that the robot called 'lycra' has all relative URLs starting with '/' disallowed. # Because all relative URL's on a server start with '/', this means the entire site is closed off. # # The third paragraph indicates that all other robots should not visit URLs starting with /tmp or /log and ignore the # individual page 'foo.html'. Note the '*' is a special token, meaning "any other User-agent"; you cannot use wildcard # patterns or regular expressions in either User-agent or Disallow lines. # # Two common errors: # Wildcards are _not_ supported: instead of 'Disallow: /tmp/*' just say 'Disallow: /tmp'. # You shouldn't put more than one path on a Disallow line (this may change in a future version of the spec) # #################################################################################################################### #################################################################################################################### # # robots.txt for www.essentialtherapeutics.com is.isaka.net # (for hosting at AZC.com - created 10/23/01) # # keep everyone out of following dirs # User-agent: * Disallow: /Access Disallow: /bin Disallow: /cgi-local Disallow: /Config Disallow: /Count Disallow: /dev Disallow: /development Disallow: /etc Disallow: /icons Disallow: /images Disallow: /Private Disallow: /pub Disallow: /Real Disallow: /Responses Disallow: /Stats Disallow: /Store Disallow: /usr Disallow: /Wrdb # # hide following individual pages # Disallow: /anim_home.js Disallow: /cush.css Disallow: /cush.js Disallow: /frame.html Disallow: /frameloader.html Disallow: /frameloader.js Disallow: /robots.txt Disallow: /scrollbox.js Disallow: /spacer_white.html Disallow: /spacer_white_loader.html # # end