A Dynamic Next.js Sitemap for a Headless CMS

A sitemap can return valid XML and still be operationally broken. The file may load at /sitemap.xml, contain the homepage, and pass a validator while omitting weeks of recently published articles.

That happened on this site because the application and the CMS have different release cycles. Next.js deployed the sitemap route, while Marble CMS continued publishing posts independently. The sitemap had a five-minute revalidation setting, but the production route remained stale and a webhook invalidated the path using the wrong semantics.

The fix required treating the sitemap as a live view of public content rather than a static deployment artifact.

Define the canonical inventory first

Before changing caching, we listed the URLs that should and should not appear.

The public inventory includes the homepage, service pages, products, the CapInsta overview, About, Contact, legal policies, editorial standards, the author page, the blog index, reviewed articles, and non-empty blog categories. Administrative, authentication, preview, API, internal search, and empty taxonomy URLs are excluded.

This inventory is also useful outside the sitemap. It exposes orphan pages, inconsistent canonicals, and navigation gaps. If a page is important enough for the sitemap but cannot be reached from a relevant navigation or content link, its internal discovery path needs work.

All generated URLs use one origin: https://www.huygenstudios.com. The sitemap does not mix HTTP and HTTPS, or the root domain and www. A separate subdomain such as CapInsta should maintain its own sitemap; the main domain links to a canonical product overview on the main site instead of claiming the subdomain URL as part of its sitemap.

Request fresh CMS data

The sitemap route calls the CMS posts list with a no-store request. This is deliberate. A normal blog page can tolerate a short cache window; the sitemap is small and should reflect the current public inventory whenever a crawler requests it.

The CMS fetch normalizes each post, accepts only published records, encodes the canonical slug, and filters anything awaiting editorial review. A CMS response that cannot be normalized is logged and omitted instead of producing a malformed URL.

For each accepted article, the sitemap uses updatedAt when available and falls back to publishedAt. Invalid timestamps fall back to a known site-update date. Categories are added only when at least one accepted article belongs to them, preventing empty archive pages from entering the index.

Opt out of metadata-route caching explicitly

In the current Next.js version used by this project, metadata routes are cached unless configured otherwise. The sitemap therefore declares:

export const dynamic = "force-dynamic";
export const revalidate = 0;

The CMS request also uses cache: "no-store". The two controls address different layers: route output and upstream data. Relying on only one made the system harder to reason about.

This is also why framework-version documentation matters. Cache APIs and invalidation profiles have changed between Next.js releases. Code copied from an older example may compile while preserving stale behavior.

Revalidate immediately after CMS changes

Marble or the publishing workflow sends a signed request to the application’s revalidation endpoint after a publish, update, deletion, or slug change. The endpoint authenticates a private header or query secret and then invalidates:

• the blog index;
• the sitemap path;
• the shared Marble posts tag;
• the individual article path and article tag when a slug is supplied;
• and explicitly supplied old paths after a URL change.

For external webhooks, the tag is expired immediately with { expire: 0 }. A stale-while-revalidate profile can intentionally serve the old value once more, which is useful for normal browsing but wrong when an external publishing system has just declared that content changed.

Slug changes need special handling. The webhook should send both the new slug and the old article path. The application can then invalidate both. The old URL should either redirect permanently to the new canonical URL or return a real 404 if no replacement exists; it should not keep serving duplicate content.

Keep robots and canonical signals aligned

The robots file references the sitemap’s production URL and allows public pages. It blocks administrative and utility areas rather than trying to use robots rules as a substitute for noindex.

Every page emits a self-referencing canonical. Article canonicals are created from normalized slugs, category canonicals use the registered category slug, and the product overview stays canonical to the main domain. This alignment matters because a sitemap is only one signal. A sitemap URL that points to a page canonicalized elsewhere creates ambiguity instead of helping discovery.

Pages awaiting editorial review remain accessible for editors and existing links, but emit noindex, follow and do not appear in the sitemap or category collections. After review, the same quality predicate allows them back into all discovery surfaces.

Verify the generated output, not just the source

The final test requests the production build’s actual routes. It checks that robots.txt and sitemap.xml return 200 responses with the expected content types, parses every <loc>, rejects foreign hosts, and confirms the presence of products, trust pages, and categories.

For a CMS-enabled environment, the test also compares the latest reviewed CMS publication timestamp with the newest article in the sitemap. Search Console should then be used to submit the sitemap and inspect representative URLs.

The durable pattern is simple: keep one canonical inventory, fetch external content deliberately, invalidate with the cache semantics of the installed framework version, and verify the output that a crawler actually receives.