The robots.txt file is used to tell web crawlers what is OK to crawl, and what is NOT ok.  This is a text file that lives in the root folder of your website.  You can add rather cryptic directives to this file to tell crawlers what path to stay out of: