When a shopper asks ChatGPT or Perplexity what to buy, an AI crawler has to fetch and read your store before it can recommend you. Many stores block these crawlers without knowing it. Here's exactly what the crawlers fetch, what silently blocks them, and how to test your store in about 60 seconds.
No signup. We read your public storefront the same way the crawlers do — nothing is stored.
Each AI company runs more than one crawler, and they do different jobs. Broadly there are three roles: a training crawler (collects pages to train future models), a search-index crawler (builds the index the assistant searches when answering), and a user-triggered fetcher (visits your page live because a user just asked about it). You can allow or block each one separately in robots.txt — and the one you block decides what you disappear from.
| Company | User agent | What it does | robots.txt |
|---|---|---|---|
| OpenAI | GPTBot | Collects public pages that may be used to train OpenAI's models | Respected |
| OpenAI | OAI-SearchBot | Builds the index behind ChatGPT search — blocking it removes you from ChatGPT search results | Respected |
| OpenAI | ChatGPT-User | Fetches a page live when a ChatGPT user asks about it | Separate token you can target |
| Anthropic | ClaudeBot | Collects public pages that may be used to train Anthropic's models | Respected |
| Anthropic | Claude-SearchBot | Indexes content to improve Claude's search results | Respected |
| Anthropic | Claude-User | Fetches a page live when a Claude user asks about it | Respected |
| Perplexity | PerplexityBot | Surfaces and links sites in Perplexity results; Perplexity says it is not used for model training | Respected, per Perplexity's docs |
| Perplexity | Perplexity-User | Fetches a page live when a Perplexity user asks about it | Perplexity says it "generally ignores" robots.txt for user-initiated visits |
The practical takeaway: blocking GPTBot does not hide you from ChatGPT answers — that's the training crawler. It's OAI-SearchBot and ChatGPT-User that decide whether ChatGPT can find and read your store when a shopper asks. Plenty of stores blanket-block every AI user agent to "protect content" and unknowingly remove themselves from AI shopping answers entirely.
Nearly every AI-readability failure traces back to one of these four.
The most common blocker — sometimes deliberate, often a leftover from a "block AI bots" tutorial. Shopify ships a sensible default robots.txt, and themes can override it via a robots.txt.liquid template (Shopify's editing guide). If a past developer added Disallow: / under any AI user agent, that engine's compliant crawlers stopped reading you that day.
A firewall can return a 403 to AI crawlers even when your robots.txt welcomes them. This is now the default posture for part of the web: since July 1, 2025 Cloudflare blocks AI crawlers by default for new customers (MIT Technology Review coverage). If your store or its proxy sits behind a bot-management layer, you have to check what it actually serves these user agents — not just what robots.txt says.
While a Shopify store's password protection is on, every storefront URL returns the password page. To an AI crawler (and to Google) your catalog simply doesn't exist. Expected pre-launch — but it also means a "coming soon" period builds zero AI visibility, so re-test the day you remove the password.
An analysis by Vercel and MERJ of hundreds of millions of crawler fetches found no evidence that OpenAI's, Anthropic's, or Perplexity's crawlers execute JavaScript — they read the raw HTML only. Shopify themes render core product content server-side, which works in your favor. But anything an app injects in the browser — reviews widgets, size guides, upsell blocks, FAQ accordions built client-side — is invisible to AI crawlers.
Paste your URL into our free checker. It fetches your storefront the way the crawlers do and reports whether the AI user agents above are blocked, plus schema, llms.txt, and answer-readiness — in about a minute, no signup.
Open yourstore.com/robots.txt in a browser and search the page for GPTBot, OAI-SearchBot, ClaudeBot, and PerplexityBot. A healthy, AI-readable setup either doesn't mention them (no rule = allowed) or explicitly allows them:
User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: /
Disallow: / under any of these means that crawler is locked out. Our free llms.txt generator also builds a companion llms.txt file — an emerging, still-optional convention that offers AI systems a clean map of your store.
On a product page, right-click and choose View page source (not Inspect), then search for a sentence from your product description. If it's in the raw source, AI crawlers can read it. If it only appears on the rendered page, it's injected by JavaScript — and per the evidence above, invisible to them.
If your domain sits behind Cloudflare or another bot-management layer, review its AI-crawler settings — a firewall 403 overrides a welcoming robots.txt. And if your store is password protected, everything above is moot until launch day.
Being readable is necessary, not sufficient. It's the gate, not the ranking.
We'll be straight about what nobody outside these companies knows yet: none of the AI engines publish how often they recrawl a given store, or the exact weight crawlability carries versus content quality when their assistants pick products to recommend. This whole surface is early and moving fast — anyone promising you "rank #1 in ChatGPT" is guessing. What is certain is the negative case: a store these crawlers can't fetch, or can't parse, can't be recommended. Fix readability first, then compete on content and structured data like you always have.
ChatGPT reaches your site two ways: OAI-SearchBot indexes it for ChatGPT search, and ChatGPT-User fetches a page live when a user asks about it. If your robots.txt or a bot-blocking firewall blocks those user agents, or your key content only appears after JavaScript runs in the browser, ChatGPT can't read it. Our free AI-visibility check tests this in about a minute.
Not by itself. GPTBot is OpenAI's training crawler. Visibility in ChatGPT search is governed by OAI-SearchBot, and live page visits by ChatGPT-User. Some stores block GPTBot for training reasons while leaving the other two open — that setup keeps you visible in ChatGPT answers.
The best public evidence says no. A Vercel and MERJ analysis of hundreds of millions of fetches found no sign of JavaScript execution by OpenAI's, Anthropic's, or Perplexity's crawlers. Shopify renders core product content server-side, which helps — but content injected in the browser by apps (reviews widgets, upsell blocks) is invisible to these crawlers.
No. While password protection is on, every storefront URL returns the password page, so AI crawlers (and Google) see no products at all. That's expected before launch — just remember to remove the password at launch and re-test.
Run the free check to see exactly which AI crawlers can — and can't — read your store. If you're on Shopify, the Hatchloop app (currently in review for the Shopify App Store) will keep the fixable gaps closed automatically once it launches.