How to add robots.txt file to a SharePoint site

1. Create a text file robots.txt with following format
   User-Agent: *
   Disallow: /_Layouts/
   Disallow: /search/
   Disallow: /Documents/Forms/
2. Add text file robots.txt to the root of the site.
3. On the top link bar of the Central Administration Web site, click Application Management.
4. On the Application Management page, in the SharePoint Web Application Management section, click Define managed paths.
5. On the Define Managed Paths page, select the correct web application.
6. On the Select Web Application page, click the web application for which you want to define managed paths.
7. Under Add a New Path, enter the following: /robots.txt
8. In the Type list, select Explicit inclusion and click OK.
9. Right-click on robots.txt and set "Everyone" Read permissions to the file
10. Reset IIS