RankEngineRankEngine Help

Sitemap URL audit

Lives on SEO Tools → Content tab → Audit sub-tab. Self-service catalog audit that doesn't require Google Search Console.

How it works (phase 703 — sitemap-index aware)

  1. Defaults the sitemap URL to https://{your-shop}/sitemap.xml — Shopify generates this automatically.
  2. Server-side fetches the XML (8MB cap, 10s timeout, SSRF-guarded).
  3. If <sitemapindex> is detected (Shopify default), recursively fetches up to 8 sub-sitemaps (sitemap_products_1.xml, sitemap_collections_1.xml, etc.) and aggregates URLs + lastmods across the whole tree.
  4. Strips <sitemap> wrappers before extracting <loc> so sub-sitemap URLs aren't HEAD-checked as page URLs (pre-phase-703 bug — fixed).
  5. Parses up to 50 <loc> entries (real page URLs from the expanded tree).
  6. Parallel HEAD-checks in batches of 8 with redirect: 'manual' so the audit measures THIS URL's status (not its destination).
  7. Aggregates: 2xx OK / 3xx redirects / 4xx-5xx broken / connection errors.

Freshness signal (phase 702/703)

Above the URL status table:

Sitemap freshness Newest: 3d ago (May 14, 2026) · Oldest: 412d ago (Apr 1, 2025) · Coverage: 1,247 of 1,247 URLs have lastmod Expanded 3 sub-sitemaps (sitemap-index detected).

The newest-lastmod color-codes: green (< 30d), amber (< 90d), red (> 90d). When red, a critical banner explains: "Sitemap hasn't been regenerated in N days. Google likely deprioritized recrawl frequency — new products may take weeks to index."

Dashboard pending pill (phase 704)

When newest lastmod > 90 days, the Dashboard shows an orange 🗺️ Sitemap stale (Nd) pill in the pending row. Click → land on the Audit sub-tab to re-probe and investigate.

Nightly probe worker (phase 705)

Fires at 08:00 UTC daily for every installed shop. Refreshes sitemap_freshness_cache without merchant action — so passive merchants still see the regression on Dashboard within 24h of any sitemap change.

Weekly digest (phase 706)

If newest lastmod > 90 days when the digest is built, section 9 surfaces:

🗺️ Sitemap stale · Nd since last <lastmod> Your sitemap's newest lastmod is {date} — N days ago. Google uses lastmod to decide recrawl frequency; staleness this long signals "site has stopped changing", which deprioritizes recrawl. New products may take weeks to index.

The headline also gets a sitemap Nd stale contribution.

Send to 404 Hunter

When 4xx/5xx issues exist, two buttons appear in the stats row:

  • Send N → 404 Hunter (primary) — writes the broken URLs to the Hunter's textarea via sessionStorage + CustomEvent and scrolls to the Hunter section.
  • Copy as text (secondary) — clipboard path for power users.

The 404 Hunter then runs HEAD-checks + AI-embedding similarity search to suggest the closest live page as a redirect target. Each suggestion has its own Apply 301 button, plus a "Bulk apply N ≥80% matches" CTA.

Limitations

  • 50-URL cap per HEAD-check audit (sitemaps are typically segmented; the first 50 surface the most broken paths)
  • 8-sub-sitemap cap on index expansion (covers Shopify's typical 3-5 sub-sitemaps; larger catalogs need GSC for complete coverage)
  • HEAD-only (doesn't validate page content, just status)
  • 6s timeout per URL, 8s for the top sitemap, 6s per sub-sitemap

Plan

Free-tier accessible. No external search-data or AI usage.

Pairs with

  • 404 Hunter — receives the broken-list, suggests semantic redirects, one-click 301 apply
  • AI Crawler Access Matrix — parallel "is robots.txt blocking AI bots from finding these URLs at all?" diagnostic