Large sites should structure their sitemap as a sitemap index pointing at multiple child sitemaps, each holding up to 50,000 URLs or 50MB uncompressed, and reference the index from robots.txt so every crawler — search and AI — can discover the full URL surface in one round trip.
The 50,000-URL and 50MB limits are set by the sitemaps.org protocol and reiterated in Google's large-sitemap documentation. When a site exceeds either, split into multiple sitemaps and expose a <sitemapindex> file that lists them.
Practical structure for a content site with tens of thousands of pages: one child sitemap per content type (articles, answers, products, categories), each capped at 45,000–50,000 URLs, with a lastmod on every entry driven by the row's real modification date. Regenerate on write, not on a batch schedule — a stale sitemap wastes crawl budget.
Reference the sitemap index from robots.txt with a Sitemap: line — Google's robots.txt guide and RFC 9309 both document the directive, and most AI crawlers respect it too.
Never list URLs in a sitemap that return 404 or a soft-404 — Google's guidance is explicit that a sitemap listing broken URLs erodes crawl trust. Test with the Search Console Sitemaps report and with a random-sample fetch from your production origin.