RankEngineRankEngine Help

llms.txt and agent.json — making your store AI-readable

Search engines have crawled your robots.txt for 30 years. AI agents (ChatGPT, Perplexity, Claude, Google's SGE) don't follow robots.txt — they look for newer signals.

RankEngine generates and serves the two files AI agents care about most.

llms.txt

A markdown manifest at /pages/llms-txt (also exposed at /llms.txt via a Cloudflare Worker on Pro plans). Tells AI agents:

  • What your site is about
  • Which products/pages are most important
  • Per-product summaries optimized for AI citation
  • Links to canonical schema.org JSON-LD

RankEngine generates llms.txt based on:

  • Your shop name + description
  • Top 50 products by SEO score
  • Active blog posts
  • Your LocalBusiness schema (if set in Local SEO)

It refreshes every time a product/page changes — no manual edits.

agent.json

A machine-readable manifest at /.well-known/agent.json. Describes:

  • Your business identity
  • Available capabilities (e.g. "view products", "check stock")
  • Contact endpoints
  • Schema.org Organization reference
  • Public API endpoints (if any)

This is the file ChatGPT's "agent mode" reads when it visits a domain to decide whether to engage.

How to enable

  1. Open /agent-ready → the Overview tab is where setup lives (Phase 133 unified Agent Ready + AI Visibility into one page with three tabs: Overview / AI Platform Queries / Manual Tasks).
  2. The page shows a per-check list with the state of each module. Items already in place show ✓ verified; items needing setup expose a "Generate" or "Apply" button.
  3. Click Apply All at the top to run every auto-fixable check at once. The cost preview shows the estimate before you confirm.
  4. Verify post-deploy by visiting /pages/llms-txt and /.well-known/agent.json on your storefront.

The merchant-facing modules:

  • llms.txt — published as a Shopify Page at /pages/llms-txt (handle llms-txt); also written as a theme asset for direct fetches.
  • ai.txt — at /pages/ai-txt, AI training/crawling policy.
  • agent.json — at /.well-known/agent.json via the storefront app proxy (Phase 102).
  • Schema.org JSON-LD — written to shop.metafields.seo.jsonld so the theme app embed emits it on every page (Phase 254 wired the metafield write so the merchant's storefront actually receives it).
  • AI-crawler robots additions (GPTBot, ClaudeBot, PerplexityBot, etc.) — written to templates/robots.txt.liquid. Phase 264 split this from update_llms_txt so merchants can grant one without the other.
  • Web Bot Auth (Pro plan) — RFC 9421 server signing for outbound API calls (Phase 102). Manage signing keys at /agent-ready → API Keys.

What about robots.txt?

RankEngine respects existing robots.txt rules. We do NOT override them. If you've blocked GPTBot or PerplexityBot in robots.txt, those agents will still respect that — and your AEO score reflects that decision.

Why this matters

Stores without llms.txt get cited ~3-5× less often in AI search results vs comparable stores with one (RankEngine's beta-merchant data, n=12). The gap is widening as AI search adoption grows.

Related