The short answer
llms.txt is a Markdown file you place at the root of your website that tells AI systems what your site is about and which pages matter most. Where robots.txt tells crawlers what they may not read, llms.txt tells AI assistants what they should read — in a clean text format that skips the navigation menus, popups, and scripts that make normal web pages hard for a language model to digest.
The standard was proposed on September 3, 2024 by Jeremy Howard, co-founder of Answer.AI and fast.ai. His argument: AI models have limited context windows, and a typical HTML page wastes most of that space on markup. A single curated Markdown file lets an AI system understand a whole site in one read. The full specification lives at llmstxt.org.
One thing llms.txt is not: an access-control file. It cannot block an AI crawler, and it cannot force one to visit. It is a guide, not a gate.
What an llms.txt file looks like
The spec is deliberately simple. A valid file is ordinary Markdown with this structure:
- An H1 heading with your site or store name — the only required part
- A blockquote with a one-or-two-sentence summary of what you sell and who for
- Optional paragraphs with extra context
- H2 sections containing lists of links to your key pages, each with a short note
- An optional section titled “Optional” for secondary links an AI can skip when short on context
Here is what that could look like for an example store:
# Example Candle Co.
> Hand-poured soy candles made in small batches.
> We ship across the US and EU; bestsellers are the
> Amber & Oak and Sea Salt collections.
## Collections
- [Bestsellers](https://example.com/collections/bestsellers): our
ten most-ordered candles
- [Amber & Oak](https://example.com/collections/amber-oak): woody,
warm scents for autumn
## Policies
- [Shipping](https://example.com/pages/shipping): US 2-4 days,
EU 5-8 days
- [Returns](https://example.com/pages/returns): 30-day returns,
unused items
## Optional
- [Our story](https://example.com/pages/about): founder story and
studio photosThat is the whole format. No XML, no schemas, no tooling required — you could write one in a text editor in ten minutes, or use our free llms.txt generator to build one from your store URL.
llms.txt vs robots.txt vs sitemap.xml
These three files get confused constantly. They do different jobs:
| File | What it does | Format | Can it block bots? |
|---|---|---|---|
| robots.txt | Tells crawlers which URLs they may or may not fetch. This is where you allow or block GPTBot, ClaudeBot, PerplexityBot, etc. | Plain-text directives | Yes (for bots that comply) |
| sitemap.xml | Lists every indexable URL so search engines can find them all. Exhaustive, not curated. | XML | No |
| llms.txt | Gives AI systems a curated, human-written summary of what matters on your site and why. | Markdown | No |
Practical takeaway: if your goal is to control AI crawlers, that is a robots.txt job. llms.txt is about being understood once a crawler or agent arrives. Our free AI-crawler check shows which AI bots your robots.txt currently allows or blocks.
Who actually reads llms.txt today? (the honest part)
This is where most articles oversell, so let us be precise about what is documented as of August 2026:
- Google does not use it. Google's John Mueller compared llms.txt to the old keywords meta tag — a self-declared claim search engines learned not to trust — and noted that no major AI service had announced support for it.
- Crawler pickup is measurably low. One agency published server-log data from 83 websites over 12 weeks: OpenAI's crawlers fetched robots.txt thousands of times but llms.txt only 7 times, and Perplexity's crawler never requested it. That is one study, not the whole web — but it matches what Mueller said about server logs generally.
- Where it is genuinely used: developer-tool ecosystems. Documentation platform Mintlify auto-generates llms.txt for every docs site it hosts, and coding assistants and agents that fetch pages on demand can consume the file when pointed at it. The pattern that works today is on-demand reading by agents, not routine crawling.
- And notably: Shopify itself adopted it. More on that next, because it changes the question for store owners.
Our honest read: llms.txt is an early-stage standard, not a proven ranking lever. Nobody can honestly promise that adding one will get you cited by ChatGPT next week. The case for it is different: it costs minutes, carries zero SEO risk, and positions you for the agent-driven shopping traffic that platforms — including Shopify — are actively building toward.
Why Shopify now generates one for every store
In April–May 2026, Shopify quietly began serving auto-generated /llms.txt and /agents.md files on storefronts — without a changelog entry, which surprised developers on Shopify's own forums. Shopify staff later confirmed the behavior is intentional: requests to /llms.txt redirect to /agents.md, which Shopify now treats as “the primary storefront discovery file for agentic and AI experiences,” and merchants can override the default by adding their own template (e.g. agents.md.liquid) in their theme.
Read that signal for what it is: the largest e-commerce platform decided every one of its storefronts should have an AI-readable discovery file, because it is building for a world where AI shopping agents browse and buy on shoppers' behalf. The catch is that the auto-generated default is generic — it identifies your store but does not showcase your bestsellers, your collections, your shipping promise, or anything that would make an agent pick you. If an agent reads the same boilerplate on your store and your competitor's, the file does nothing for you.
Should your store add (or customize) one?
Our recommendation, given everything above:
- Fix access first. Before curating content for AI, make sure AI crawlers can reach your store at all — a robots.txt that blocks GPTBot or PerplexityBot makes everything else moot. Our free AI-visibility check reads this from your live store in about a minute.
- Then publish a real llms.txt. Use the free generator or write one by hand: name, honest summary, your top collections and policy pages with one-line notes. On Shopify, add it as a theme template so it overrides the generic default.
- Do not stop there. The signals with stronger evidence behind them today are structured data (Product, Offer, FAQ schema), real product descriptions, and crawlable pages. llms.txt is a cheap addition to that foundation — not a substitute for it.
Time cost: about ten minutes. Risk: none we are aware of — engines that ignore the file simply ignore it. Upside: your store is legible to the agents that do read it, on a web that is clearly moving their way.