Sitemaps and robots.txt

Two small files sit at the root of almost every website and decide what Google is allowed to look at and what it knows exists. They are the plumbing of SEO: nobody notices them until one is wrong, and then nothing else works.

What each file is for

They are often mentioned in the same breath, which hides the fact that they do opposite jobs.

robots.txt is a plain text file at yoursite.com/robots.txt that tells automated visitors — search engine bots — which parts of your site they should not request. It is a "please don't go here" sign. It is a request, not a lock: it does not password-protect anything, and anyone can read the file.

sitemap.xml is a machine-readable list of the pages you do want found. It is a "here is everything" note. It helps Google discover pages, particularly ones that few links point to.

For a small bakery site, robots.txt might ask bots to skip the checkout and account pages, while the sitemap lists the homepage, the menu, the opening hours and every blog post.

How they actually behave

The most important thing to understand about robots.txt is what it does not do. Blocking a page stops Google fetching it — it does not reliably keep the page out of search results. If other sites link to that address, Google can still list it, having never read it. To genuinely keep a page out of results you use a noindex tag on the page itself, and for Google to see that tag it has to be allowed to crawl the page. Blocking a page in robots.txt and adding noindex to it at the same time cancels the noindex.

Sitemaps are gentler. A sitemap is a hint, not an instruction. Listing a page does not guarantee Google will crawl it, and certainly does not guarantee indexing — Google decides that separately. Equally, a page missing from your sitemap can still be found and indexed through ordinary links.

Both files are optional. A small, well-linked site can rank perfectly well with neither.

What to do

• Visit yoursite.com/robots.txt in your browser right now and read what is there. Many site builders generate one automatically, and it is not always what you would choose.
• Add a link to your sitemap inside robots.txt, on its own line: Sitemap: https://yoursite.com/sitemap.xml.
• Submit the sitemap once in Google Search Console, under Sitemaps. Search Console will then report how many of its URLs were discovered.
• Keep the sitemap generated automatically, not written by hand — a hand-written list goes stale the day you publish something new.
• List only pages you want indexed. A sitemap full of redirects, error pages and noindex pages sends mixed signals.

Common mistakes

Shipping a development block to production. Sites under construction are often set to block all bots. That two-line file getting copied to the live site is one of the most common catastrophic SEO mistakes there is, and the site looks entirely normal while it happens.

Using robots.txt to hide private content. The file is public and lists exactly what you did not want visited. Anything genuinely private needs a login, not a polite request.

Expecting a sitemap to fix ranking. A sitemap helps Google find pages. It has nothing to do with whether they deserve to rank.

Sources: Introduction to robots.txt and Learn about sitemaps.