RankEngineRankEngine Help

AI Crawler Access Matrix

Lives at Content Studio → Audit sub-tab → AI Crawler Access Matrix.

The Agent-Ready scan has a high-level "AI bot rules" check that passes if any of 6 AI bots are mentioned in your robots.txt. That's a useful baseline but hides a critical trap: a robots.txt containing

User-agent: GPTBot
Disallow: /

passes the green check (GPTBot is mentioned) while being explicitly blocked from ChatGPT. Your site is invisible to that crawler despite the dashboard green badge.

This panel surfaces the per-bot truth.

How it reads

For each of 15 AI bots, you'll see one of three statuses:

Status Meaning
allowed Explicit Allow: / block, OR a UA block with non-catch-all rules (paths only)
disallowed Disallow: / with no matching Allow: / — bot is fully blocked
default No UA block exists — falls through to the * wildcard policy

A red banner appears at the top if any bot is disallowed.

Wildcard policy block (phase 756)

The matrix also surfaces the User-agent: * (wildcard) policy as a separate summary block above the per-bot table. This is the default-for-unlisted-bots baseline.

  • Disallow: / (no Allow override): red badge — every bot NOT explicitly named in robots.txt is blocked
  • Allow: /: green badge — every bot NOT explicitly named is permitted
  • No catch-all rules: info badge — wildcard exists but has only path-specific rules (most common Shopify default)
  • Disallow paths + Allow paths: per-path chips (e.g. Disallow: /admin, Disallow: /checkout)

This closes the "what does 'default' mean for this unlisted bot?" interpretability gap that the per-bot table alone couldn't answer.

The 15 bots tracked

GPTBot, ChatGPT-User, Google-Extended, CCBot, anthropic-ai, Claude-Web, ClaudeBot, Bytespider, PerplexityBot, cohere-ai, Applebot-Extended, Amazonbot, Meta-ExternalAgent, FacebookBot, Omgilibot, Diffbot.

(Same list the Agent-Ready scanner probes — see lib/agent-ready/robots-builder.ts.)

Parser semantics

robots.txt blocks are tokenized as (User-agent: X)+ (Allow/Disallow: Y)+. A blank line OR a new User-agent: after rules ends the current block. Multiple consecutive UAs share the same rule set.

The classifier:

  1. Looks up the bot's UA block case-insensitively.
  2. If found:
    • Disallow: / AND no Allow: / → disallowed
    • Any Allow: / → allowed
    • Any other rules (e.g. Disallow: /admin) → allowed (catch-all not blocked)
  3. If not found, falls back to the * wildcard:
    • * has Disallow: / → disallowed
    • Otherwise → default

Domain override

By default the panel probes your shop's primary domain. The Re-probe input accepts any hostname so you can audit a competitor's robots.txt or test a staging domain.

What to fix

If any bot shows disallowed, edit your robots.txt (Shopify: Online Store → Themes → Edit code → templates/robots.txt.liquid). Remove the Disallow: / line under that UA block, or add an explicit Allow: /.

Related

  • Agent-Ready scan (/agent-ready) — the broader AEO audit. This panel zooms in on the bot-access dimension.
  • AI Visibility · Dashboard card — shows whether AI assistants mention you. The crawler matrix shows whether they can crawl you in the first place.