Module 3 · 3.4

sitemap.xml: the table of contents for search engines

Module 3: The technical basics of a page · Reading time: 11 minutes

Learning objective: After this lesson you can read a sitemap.xml, judge whether it is correct and make sure search engines find it.

Key concepts: XML sitemap, sitemaps.org protocol, urlset, url, loc, lastmod, changefreq and priority (ignored by Google), limit of 50,000 addresses and 50 MB, sitemap index, canonical addresses, Sitemap: line in robots.txt, automatically generated sitemap

What a sitemap is, and what it isn't

A sitemap is a file listing the addresses on your site that you want to see in the search results. It is the table of contents you hand to a search engine. A crawler finds pages mainly by following links (lesson 1.2). A sitemap helps with that: it also lists pages that few links point to, and it tells the search engine when a page last changed.

There are two things a sitemap doesn't do:

It isn't a guarantee. Google says so itself: a sitemap doesn't guarantee that everything in it will be crawled and indexed. Submitting a sitemap is a hint.
It doesn't replace good links. A page that only appears in the sitemap, with not a single link to it from your own site, tells a search engine that you don't consider it very important yourself.

Do you need a sitemap? According to Google, it helps mostly with a large site, a new site with few links from elsewhere, and a site with lots of video or images. A small site (Google mentions roughly 500 pages or fewer) with good internal linking can do without. But a sitemap costs next to nothing and most website builders create one automatically. If you have one, make sure it is correct: a sitemap full of errors is worse than no sitemap at all.

What a sitemap looks like

The standard sitemap is an XML file following the sitemaps.org protocol, which Google, Bing and other search engines all use. This is what the sitemap of Bloemenatelier Lotte looks like:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.example.com/</loc>
    <lastmod>2026-11-02</lastmod>
  </url>
  <url>
    <loc>https://www.example.com/bruidsboeketten</loc>
    <lastmod>2026-10-14</lastmod>
  </url>
  <url>
    <loc>https://www.example.com/contact</loc>
    <lastmod>2026-06-30T09:15:00+02:00</lastmod>
  </url>
</urlset>

Element	Required?	What it does
`<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">`	yes	The root element, with exactly this namespace
`<url>`	yes	One block per address
`<loc>`	yes	The full address: with `https://` and the host name, shorter than 2,048 characters
`<lastmod>`	no	When the content of the page last really changed, in W3C date format: `2026-11-02` or with time and time zone
`<changefreq>`	no	"How often does this change?" Google ignores it
`<priority>`	no	A number from 0.0 to 1.0. Google ignores this too

A few rules that often go wrong:

Full addresses. Not /contact, but https://www.example.com/contact. Google explicitly asks for full, absolute addresses.
UTF-8 and escaping. The file must be UTF-8, and you write characters such as & in an address as &. A bare & makes the whole XML invalid.
Same scheme and same host. According to the protocol, a sitemap may only contain addresses with the same scheme and the same host as the sitemap itself. So a sitemap on https://www.example.com/ with addresses on http://example.com/ is wrong. It is also a sign that your canonical and your redirects aren't aligned (lessons 3.5 and 4.1).

lastmod: only if it's accurate

Google uses <lastmod> if it is "consistently and verifiably" accurate. In 2023 Google added what counts as a change: the main text, the structured data or the links on the page. A new year in the footer isn't a change. If you put today's date on every page every day, Google learns that your dates mean nothing and ignores them. An honest date, or no date, is better than a made-up one.

Large site? Use a sitemap index

A single sitemap may contain at most 50,000 addresses and be at most 50 MB in size (uncompressed). If your site is bigger, or you want to split the sitemap up (pages, blog, products), you create a sitemap index: a sitemap of sitemaps.

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>https://www.example.com/sitemap-paginas.xml</loc>
  </sitemap>
  <sitemap>
    <loc>https://www.example.com/sitemap-blog.xml</loc>
  </sitemap>
</sitemapindex>

You then submit only the index. WordPress works this way: https://www.example.com/wp-sitemap.xml is an index that points to separate sitemaps for pages, posts, categories and authors. SEO plugins often replace it with their own index, for example sitemap_index.xml.

Which addresses belong in it?

Only the addresses you want to see in the search results, and then in their canonical form (lesson 3.5). So not:

addresses that redirect (the http:// variant, the old name of a page);
addresses that return a 404 or 410;
pages with noindex (lesson 1.3): you'd be saying "include this" and "don't include this" at the same time;
pages you block in robots.txt (lesson 3.3);
variants with parameters, such as ?sorteer=prijs or ?utm_source=nieuwsbrief;
thank-you pages, shopping baskets, login pages.

You'll see these contradictions in Search Console (lessons 2.2 and 2.4): in the Pages report you can filter on the addresses from your sitemap, and then you'll see, for example, addresses excluded by a noindex tag or blocked by robots.txt. Every such message is an address that doesn't belong in your sitemap, or a page that was shut off by mistake.

How search engines find your sitemap

There are two standard ways, and you should do both:

A line in robots.txt (lesson 3.3). Every search engine reads robots.txt, so even a search engine where you never created an account will find it:
```
Sitemap: https://www.example.com/sitemap.xml
```
The address must be complete. You may include several Sitemap: lines.
Submitting it in Search Console and Bing Webmaster Tools (lessons 2.2 and 2.5). You'll then also see whether fetching it worked and how many addresses were found.

In the past you could "ping" Google with a special address. Google scrapped that in 2023: such a ping no longer does anything. If you want to tell search engines about a change quickly, the way to do it is lastmod and, for Bing and a few other search engines, IndexNow (lesson 2.5).

In practice: make it yourself or let it be made

Hardly anyone writes a sitemap by hand. Since version 5.5, WordPress creates wp-sitemap.xml itself and adds the reference to its (virtual) robots.txt. When you publish, WebOké's AI Builder places a sitemap.xml in the web folder, with the date of the last change for each page, plus a robots.txt containing the Sitemap: line. Other website builders do similar things. Your job is to check:

Open /robots.txt on your own domain (for example https://www.example.com/robots.txt) and see whether there is a Sitemap: line.
Open that sitemap in your browser. Does it contain the addresses you expect, with https:// and the right host name (with or without www)?
Click a few of them. Does each page load properly straight away, without a redirect?

Summary

A sitemap is a table of contents and a hint, not a guarantee. Small, well-linked sites can do without, but a correct sitemap always helps.
Required are urlset with the right namespace, url and loc with full addresses; lastmod only if it's accurate. Google ignores changefreq and priority.
At most 50,000 addresses and 50 MB per file; above that, a sitemap index.
Only canonical addresses that return 200, without noindex and not blocked.
Put a Sitemap: line in robots.txt and submit the sitemap to Google and Bing. The old ping no longer works.

Quiz

Exercise

Exercise A3.4: Is your sitemap valid and easy to find? Enter the address of your site. The program looks for your sitemap the way a search engine does (first via robots.txt, otherwise at /sitemap.xml), checks whether it is valid XML according to the protocol, whether it is listed in robots.txt, whether the addresses are complete and on your own host, and fetches a few of them to see whether they really may be indexed. No site of your own? Use european-union.europa.eu.

Cookies at WebOké

Cookie settings

Necessary

Marketing