#%% Telling robots to bypass certain URLs
#
# This sample file is written to the "robots exclusion protocol"
# or "robots exclusion standard".  Well behaved robots (that's 
# all the important ones!) use this file to check where they are
# unwelcome ... and they should then only crawl / use your other
# pages.
#
# See
# http://en.wikipedia.org/wiki/Robots.txt
# http://www.robotstxt.org/robotstxt.html
# and checker at
# http://tool.motoricerca.info/robots-checker.phtml
#
# Why do you want to exclude certain URLs when the whole point
# possibly accessibility combinations
#
# * CGI program outputs differ every time - no point in indexing them
#

User-agent:  *
Disallow: /cgi-bin/			# Disallow cgi programs
Disallow: /blast/
Disallow: /msd/
Disallow: /454/

User-agent: TurnitinBot
Disallow: /

# Note that blank lines are NOT allowed within the block!