AI Bot Manager for Shopify: robots.txt Guide (2026)

Why AI Bot Policy Matters on Shopify

Your robots.txt file tells crawlers which URLs they may fetch. For AI search, the decision is strategic: allow retrieval bots and your products can surface in ChatGPT, Perplexity, and Google AI answers. Block them broadly and you opt out of the fastest-growing referral channel on Shopify without touching a single product page.

Shopify’s platform stance is opt-out, not opt-in. Default rules allow major bots the same way they allow Googlebot (Shopify Ranked, 2026). That is good news for merchants chasing AI shopping visibility, but it also means silence equals permission. If you never audit crawler access, you may still lose citations because Cloudflare, a theme edit, or an old Disallow: /policies/ rule blocks bots at the edge.

AI Bot Manager is not about fear of scrapers. It is about intentional policy: which engines may read your catalog, which paths stay private, and how that policy stays stable after theme updates.

The Six AI User-Agents to Know

User-agent	Operator	Primary role
GPTBot	OpenAI	Training and broad crawl
OAI-SearchBot	OpenAI	ChatGPT Search grounding
ChatGPT-User	OpenAI	On-demand fetch when users click citations
ClaudeBot	Anthropic	Claude web retrieval
PerplexityBot	Perplexity	Citation indexing
Google-Extended	Google	Gemini / AI training opt-out token

Perplexity-User fetches specific pages during live queries. CCBot (Common Crawl) feeds multiple downstream models (Surfient AI bots guide).

Critical distinction: GPTBot is training-oriented. OAI-SearchBot and ChatGPT-User power search and shopping experiences. Blocking GPTBot alone does not remove your store from every ChatGPT citation path (Shopify Ranked).

Three Policy Positions (Pick One Deliberately)

1. Allow all (default, best for most stores)

Keep Shopify defaults plus explicit Allow comments for auditability. Maximizes AI shopping and citation surfaces. Recommended when AI discovery is a growth channel.

2. Block training, allow retrieval

Disallow GPTBot, ClaudeBot, and Google-Extended while allowing OAI-SearchBot, PerplexityBot, and ChatGPT-User. Use when brand policy restricts model training but you still want shopping visibility (Shopify Ranked matrix).

3. Block all AI bots (rare for ecommerce)

Explicit Disallow: / per AI user-agent. Almost never correct for product catalogs. Occasionally justified for editorial brands with proprietary content and no AI channel ambition.

How to Edit robots.txt.liquid on Shopify

You cannot edit robots.txt as a static file. Shopify generates it from templates/robots.txt.liquid (Shopify Help Center).

Rules:

1. Keep Shopify’s default Liquid loop so platform updates still apply.
2. Add custom user-agent blocks below the default output.
3. Never replace the template with plain text only.
4. Disallow /cart, /checkout, /account, and /admin for all bots (Shopify defaults already handle most of this).

Recommended allowlist block (GEO-friendly)

Add after the default loop (Surfient template):

# AI retrieval bots — explicit allow for audit trail
{%- comment -%} GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot {%- endcomment -%}

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

Named allows are redundant when User-agent: * already permits crawling, but they document intent for your team and future theme migrations.

Example block (training opt-out)

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

Leave PerplexityBot and OAI-SearchBot allowed if citations still matter to you.

Verify Crawler Access (Do Not Trust robots.txt Alone)

Run from terminal:

curl -A "GPTBot" -I https://yourstore.com/products/example
curl -A "PerplexityBot" -I https://yourstore.com/robots.txt

Expect HTTP/2 200 on product URLs you want cited. A 403 with permissive robots.txt usually means Cloudflare Bot Fight Mode or similar CDN rules (Surfient).

Also check:

• Theme robots.txt.liquid did not add broad Disallow: /policies/ that blocks helpful policy pages AI assistants cite
• No accidental Disallow: / inside a wildcard group
• llms.txt is fetchable if you publish it (bots must reach /llms.txt)

robots.txt vs llms.txt vs Schema

Layer	Function
robots.txt	Permission to crawl URLs
llms.txt	Curated map of important URLs
JSON-LD	Typed product facts on each page

Blocking bots while publishing llms.txt sends mixed signals. Allow retrieval bots, ship llms.txt, and fix schema gaps together.

When to Re-Audit

• After every theme migration or agency handoff
• When AI referral traffic drops in GA4 week over week
• When citation tracking shows competitors replacing you on core prompts
• After enabling Cloudflare bot protection

Nexis CRO AI Bot Manager applies per-bot allow/block rules from the Shopify admin without hand-editing Liquid on every launch.

FAQ

Does Shopify block AI bots by default?

No. Add explicit Disallow rules in robots.txt.liquid to block specific user-agents.

Where do I edit robots.txt?

templates/robots.txt.liquid in your theme. Preserve Shopify's default Liquid loop.

Should I block GPTBot?

Most stores should not. Blocking training bots is a policy choice; blocking retrieval bots hurts AI shopping visibility.

Can Cloudflare override robots.txt?

Yes. Check CDN bot settings and response codes, not just robots.txt content.