XML Sitemaps Done Right (and the Errors That Undermine Them)

An XML sitemap is a list of URLs you want search engines to know about. It's simple in concept and easy to get subtly wrong in ways that slow indexing and waste crawl budget. Let's build one that actually helps.

What a sitemap does and doesn't do

A sitemap aids discovery — it tells crawlers which URLs exist and when they last changed, which is especially valuable for large sites, new sites, sites with poor internal linking, and pages with few inbound links. What it does not do:

• It doesn't guarantee indexing. Inclusion is a suggestion, not a command.
• It doesn't improve rankings directly.
• It doesn't override quality or crawl signals.

Think of it as a table of contents you hand the crawler, not a fast pass to the top of results.

The one rule that matters most

Only include canonical, indexable, 200-status URLs you actually want indexed. This single rule prevents the majority of sitemap problems. That means your sitemap must NOT contain:

• Redirecting URLs (301/302) — send the crawler on a wasted hop.
• 404 or 410 URLs — dead ends that erode trust in the sitemap.
• Non-canonical URLs — URLs that canonicalize elsewhere.
• Noindexed URLs — you're asking the crawler to index something you told it not to. Contradictory signals.
• Blocked URLs — anything disallowed in robots.txt.
• Parameter and session variants.

A sitemap full of these is a mixed-signal mess. Keep it clean.

Format and limits

• 50,000 URLs and 50MB uncompressed per sitemap file. Exceed either and split into multiple sitemaps.
• Use a sitemap index file to reference multiple sitemaps — split by section (/products, /blog, /categories) so you can diagnose indexing per content type.
• Use absolute URLs with consistent protocol and host.
• Gzip large sitemaps to save bandwidth.

The tags worth including

• <loc> — the URL. Required.
• <lastmod> — last modification date. Keep it accurate. If every URL shows today's date on every generation, crawlers learn to ignore it. An honest lastmod helps them prioritize genuinely updated pages.
• <changefreq> and <priority> — largely ignored now. Don't obsess over them; accurate lastmod is what matters.

Specialized sitemaps

Beyond the standard page sitemap, consider:

• Image sitemaps — help image discovery for visual content.
• Video sitemaps — surface video metadata.
• News sitemaps — for publishers in Google News (strict 48-hour freshness rules).

Submit and reference it

Two steps to make sure it's found:

1. Reference it in robots.txt: add Sitemap: https://example.com/sitemap.xml. Generate a correct robots.txt at /tools/robots-generator.
2. Submit it in Search Console via /search-console — this also unlocks the Sitemaps report showing discovered vs. indexed counts and any parse errors.

Keep it automatically fresh

A static, hand-maintained sitemap rots fast. Your CMS or build process should regenerate the sitemap automatically whenever content is published, updated, or removed, and update lastmod honestly. New pages should appear within minutes; deleted pages should drop out immediately.

Audit it regularly

Sitemaps drift out of sync with reality. Run these checks:

1. Sitemaps report in Search Console — compare submitted vs. indexed. A big gap signals quality, duplication, or crawl issues on those URLs.
2. HTTP Status Checker at /tools/http-status — batch-verify that sitemap URLs return 200, not redirects or errors.
3. Full Site Audit at /full-site-audit — it cross-checks your sitemap against what's actually crawlable and indexable, flagging orphaned URLs (in the sitemap but not linked internally) and included-but-noindexed conflicts.
4. Broken Link Checker at /tools/broken-link-checker — catch dead URLs before they reach the sitemap.

Common mistakes recap

• Including non-indexable or non-canonical URLs.
• Fake or unchanging lastmod values.
• Forgetting to update after content changes.
• Exceeding size limits without splitting.
• Not referencing the sitemap in robots.txt.
• Leaving deleted URLs in place.

The bottom line

A sitemap is a precision instrument for discovery: a clean, current list of exactly the canonical URLs you want indexed, regenerated automatically and audited regularly. It won't lift your rankings by itself, but it speeds up how fast your best content gets found and how quickly your changes register — and on a large site, that speed is worth real traffic.