When you launch a website, two plain text files can help search engines understand it: robots.txt and sitemap.xml. They are easy to create, and also easy to get wrong in ways that quietly hide your whole site. This guide explains what each one does and where it stops.

What robots.txt is

Robots.txt is a file at the top of your site, reachable at your-domain/robots.txt. It lists rules for automated visitors called crawlers. A rule can say "do not visit anything in /admin/" or "here is where my sitemap is". Well-behaved crawlers, including the major search engines, read this file before they crawl.

What robots.txt cannot do

  • It is not security. The file is public and only a request. Bad bots ignore it, and anyone can read it to see which folders you wanted to hide. Protect private areas with passwords, not with robots.txt.
  • It does not remove pages from search. A blocked address can still appear in results if other sites link to it, usually with no description. To keep a page out, use a noindex tag or password protection. Note that if a page is blocked in robots.txt, the crawler cannot see its noindex tag.
  • It cannot stop every AI crawler. You can ask known ones to stay away by name, and many respect that, but the list of names changes and compliance is voluntary.

The most expensive mistake

During development, people often add Disallow: / to keep a test site out of search. If that file is copied to the live site, the whole site is asked not to be crawled. Always open your live /robots.txt after launch and read it.

What a sitemap is

A sitemap is an XML list of the pages you want found, optionally with the date each changed. It helps new sites, large sites, and pages that few other pages link to. It does not guarantee indexing, and search engines ignore the priority and change-frequency fields. Include only pages you actually want in results: leave out login pages, thank-you pages and duplicates.

Try it

Choose what to allow, list any folders to keep out and add your sitemap address. Copy or download the file and upload it to the top folder of your site. Then build the list of pages with the XML Sitemap Generator.

Setup checklist

  1. Create robots.txt and open it in your browser to confirm it loads.
  2. Make sure it does not contain a blanket Disallow: / on a live site.
  3. Create sitemap.xml with only the pages you want found.
  4. Add a Sitemap: line to robots.txt.
  5. Submit the sitemap in Google Search Console and Bing Webmaster Tools.

To make each page's search result clearer too, read about title and meta description length.