RankEngineRankEngine Help

When do the background workers run?

RankEngine has 3 detection workers that run daily on a leader-locked schedule (only one Cloud Run instance fires them). They populate caches that the Dashboard pending pills row + weekly digest read.

The schedule

UTC hour Worker What it does Cache table
06:00 AI Crawler Probe Fetches /robots.txt for every installed shop; parses per-bot rules for 15 AI bots; HEAD-checks declared Sitemap: URLs ai_crawler_matrix_cache
08:00 Sitemap Freshness Probe Fetches /sitemap.xml, expands sitemap-index, aggregates <lastmod> across the tree sitemap_freshness_cache
10:00 Competitor Schema Snapshot Fetches each tracked competitor's homepage + merchant's own storefront; extracts <script type="application/ld+json"> types; writes to history table schema_snapshots

All three are deliberately spaced 2 hours apart so outbound fetch volume doesn't bunch up.

Why daily?

  • Robots.txt + sitemap.xml don't change minute-to-minute. A nightly probe is enough to catch the regression within 24h.
  • Schema snapshots are needed for the 7-day diff window — daily granularity is the right resolution.
  • Outbound fetch budget: 200 shops × 3 probes × 1-10 sub-fetches each = bounded.

Where the cached data shows up

Phases 780-781 wired cache-on-mount across the audit panels:

  • AI Crawler Matrix panel reads ai_crawler_matrix_cache on mount (instant render, no robots.txt fetch). Falls back to live probe on cache miss (first-install before the 06:00 UTC worker fires).
  • Sitemap URL Audit panel shows cached newest-<lastmod> banner above the audit input (before the merchant clicks Audit).
  • Dashboard pending pills row also reads all 3 caches in parallel via /dashboard/pending.

This means the merchant sees the data from the most recent nightly probe within milliseconds of opening a panel, without waiting for a fresh outbound fetch.

What if I want a fresh probe NOW?

  • AI crawler matrix — visit SEO Tools → Audit → AI Crawler Access Matrix and click "Re-probe". Fires the live endpoint which UPSERTs the cache AND adds the (live-only) sitemap declarations + wildcard policy fields.
  • Sitemap freshness — visit SEO Tools → Audit → Sitemap URL Audit and click "Audit sitemap". Same pattern.
  • Competitor schemas — visit SEO Tools → Spy & Compare → Competitor Schema Spy, paste a URL, click "Spy" + "Save snapshot for diff". The URL gets added to the next snapshot cycle.

Retention

Cache Retention Configurable via
ai_crawler_matrix_cache rolling (UPSERT only — no history) —
sitemap_freshness_cache rolling (UPSERT only — no history) —
schema_snapshots 90 days SCHEMA_SNAPSHOT_RETENTION_DAYS env
schema_apply_log 365 days SCHEMA_APPLY_LOG_RETENTION_DAYS env
ai_visibility_snapshots 365 days AI_VISIBILITY_SNAPSHOT_RETENTION_DAYS env
competitor_rank_history 180 days COMPETITOR_RANK_RETENTION_DAYS env
keyword_rank_history 180 days KEYWORD_RANK_RETENTION_DAYS env
webhook_event_dedupe per-row expires_at — (hardcoded TTL on write)
blog_jobs 90 days BLOG_JOBS_RETENTION_DAYS env
article_performance_history 365 days ARTICLE_PERFORMANCE_RETENTION_DAYS env
seo_optimization_plans 180 days SEO_OPTIMIZATION_RETENTION_DAYS env
seo_optimization_snapshots 180 days SEO_OPTIMIZATION_RETENTION_DAYS env
seo_optimization_batches 180 days SEO_OPTIMIZATION_RETENTION_DAYS env
search data cache per-row expires_at — (hardcoded TTL on write)
activity_log 180 days ACTIVITY_LOG_RETENTION_DAYS env
autopilot_activity 180 days ACTIVITY_LOG_RETENTION_DAYS env
sync_log 90 days SYNC_LOG_RETENTION_DAYS env

Note: api_cost_log is intentionally NOT pruned (billing audit / compliance retention).

Retention pruning happens at the end of each schema snapshot worker cycle (10:00 UTC).

How to verify a worker ran

Cloud Run logs (Cloud Console → Logs Explorer) show per-cycle stamps:

  • [AiCrawlerProbe] Probing N shop(s) at 06:00 UTC
  • [SitemapFreshness] Probing N shop(s) at 08:00 UTC
  • [CompetitorSchemaMonitor] Snapshotting for N shop(s) at 10:00 UTC

Why "leader-locked"?

Cloud Run can scale to multiple instances. Without a lock, all instances would fire the same worker every hour, multiplying outbound fetches. Phase 638+ uses pg_try_advisory_lock(42) so only one instance becomes the "leader" and runs the workers. Other instances skip.

If the leader instance crashes, the lock auto-releases and the next cycle on another instance picks up the leadership.

Related