What are sitemap.xml and robots.txt for and how do you set them up correctly?

Two text files at the root of a site, fifty lines between them. Every bot reads them — Google's and ChatGPT's alike — before opening a single page.

Their roles are different, and confusing the two produces the most common technical SEO mistake there is.

robots.txt says WHERE to go

The file lives at exactly one address: example.com/robots.txt. Anywhere else it has no effect.

User-agent: *
Disallow: /admin/
Disallow: /cart/

Sitemap: https://example.com/sitemap.xml

The thing almost nobody knows: Disallow does not mean "do not index". It means "do not fetch". The difference is practical and unpleasant — Google can still place the page in its index based on links alone, and the result is a listing with no description and a note that the content could not be read.

To remove a page from the index you need a noindex meta tag — and for Google to see it, the page must be fetchable. Blocking it in robots.txt means noindex is never read. Two measures that look similar cancel each other out.

The mistake that closes a site

User-agent: *
Disallow: /

Two lines, and the whole site stops being read. This is not a rare accident — it is exactly what every staging environment carries, and it travels to production along with everything else.

If traffic fell overnight, this is the first thing to open. The other causes are under why organic traffic dropped.

sitemap.xml says WHAT to read

A sitemap is a list of URLs with the date each last changed. It guarantees nothing about indexing — it only states that these pages exist and when they moved.

<url>
  <loc>https://example.com/services</loc>
  <lastmod>2026-09-19</lastmod>
</url>

Three rules decide whether Google trusts it at all.

Indexable URLs only. A noindex page, a redirect or a 404 inside a sitemap is a contradiction: the list says "read this", the page says "do not". A few such lines and trust in the whole file drops.

lastmod is real or absent. This is the most important rule and the most often broken. Google uses lastmod only when it trusts it, and identical dates destroy that trust: forty URLs sharing one day means the date is a deploy stamp, not a page history.

Canonical URLs. One page, one line. Not http and https, not with and without www, not with parameters.

priority and changefreq can be dropped: Google has ignored them for years and has said so publicly.

What else belongs in robots.txt

The Sitemap: line is the one declaration that works with no registration at all — Google and other bots both read it. If you publish an RSS feed, declare it the same way: Search Console accepts RSS as a sitemap and uses it for fast discovery of new content.

AI crawlers are controlled in the same file through separate User-agent blocks. Whether you should is a separate question, covered under should I block AI crawlers.

What to check right now

Open example.com/robots.txt in a browser. If it contains Disallow: /, stop and fix it. Open example.com/sitemap.xml and check whether every date is the same.

Declare both in Search Console — submit the sitemap manually, because that is the only way to see how many of its URLs were actually read.

The free scan checks both automatically: whether they exist, whether the sitemap matches the pages that actually resolve, whether the dates are identical, and whether robots.txt blocks anything it should not.

Related articles

The same topic from another angle — each answers its own question:

• Why Google does not index pages
• Discovered, currently not indexed: what it means
• Should I block AI crawlers

How this feeds into the score is described under what we check.

Sources

What this text is based on — primary sources, not retellings:

• Google Search Central — how Google interprets robots.txt
• Google Search Central — build and submit a sitemap
• sitemaps.org — protocol specification