This video explains custom robots files and how to create them. Robots files are used to provide web crawlers instructions on what to exclude from searches. You can either append a robots file to the default robots file or replace it.