The Definitive llms.txt Guide: What It Is, What Belongs In It, and Whether It Actually Works

Key Takeaways

• llms.txt is a plain-text file at your domain root that gives AI crawlers a curated map of your most important pages, use cases, and FAQs, the same role robots.txt plays for search crawlers, but aimed at large language models instead of search indexers.
• It is not a ranking guarantee. OpenAI's GPTBot and other AI crawlers do fetch it, but a good llms.txt is one input into citation, not a switch you flip.
• What actually moves the needle is specificity: a curated map of real pages and real use cases in plain language, not a generic template with placeholder text.
• You can generate a real one in minutes with GetIntel's free llms.txt generator, and ship it as a PR through Claude Code or Cursor rather than hand-editing it.
• Measuring whether it "worked" means re-probing the specific buyer prompts you were missing before and after, not just confirming the file is live.

---

llms.txt shows up in almost every "how to get cited by ChatGPT" checklist, and almost none of those checklists explain what should actually go in the file, or how to know if it did anything. This is the complete version: what it is, what belongs in it, how to generate and ship one without hand-writing markdown, and how to check whether it moved your actual citation rate.

What llms.txt Actually Is

llms.txt is a plain-text markdown file, placed at yourdomain.com/llms.txt, proposed as a standard way for websites to tell AI models what they are and where to find their most important content. It's modeled directly on robots.txt: same location convention (domain root), same idea (a machine-readable file that gives crawlers structured guidance), different audience. Where robots.txt tells search crawlers what they can and can't index, llms.txt tells AI crawlers and retrieval systems what your site actually contains, in a format built for a language model to read quickly rather than a format built for humans to browse.

It typically includes a short description of what the site or product is, a curated list of key pages with one-line descriptions, and enough context (use cases, who it's for, what problem it solves) that a model encountering the file for the first time can accurately summarize the product without guessing.

Does It Actually Get Crawled?

Yes, with a caveat. OpenAI's GPTBot and several other AI crawlers do fetch llms.txt as part of crawling a domain. What's less certain, and what most llms.txt content glosses over, is exactly how much weight any given model places on it relative to everything else it knows about your brand (training data, third-party mentions, live search results). Treat it as a structured, crawlable summary that removes ambiguity about what your site contains, not as a lever that overrides everything else. It's most useful as the "Foundation" layer: the baseline technical signal that makes every other AI-visibility effort (content, backlinks, entity recognition) easier for a model to parse correctly, not a replacement for those efforts.

What Actually Belongs in a Good llms.txt

The difference between an llms.txt that does something and one that's decorative is specificity. Here's the shape that holds up:

A one-paragraph description of what you are. Written the way you'd describe the product to a smart stranger, not marketing copy. "X is a tool that does Y for Z" beats a tagline every time, because a model reading this file is trying to extract facts, not vibes.

A curated list of key pages, not every page. Homepage, pricing, docs, and your two or three most important use-case or comparison pages. Five to ten entries, each with a one-line description of what's actually on that page. A 200-URL dump defeats the purpose: the value is in the curation.

A use-cases section in plain text. Who uses this, for what problem, with what outcome. This is the section most templates skip and the one that carries the most weight, since it's the part a model can lift almost directly into an answer.

FAQs, if you have them. Short, direct question-and-answer pairs mirror how buyers actually phrase queries to AI models, which makes them easier for a model to retrieve and reuse.

Here's a real shape, not a placeholder:

# GetIntel
> AI search visibility scoring + remediation for SaaS founders. We
> score your Findability across 5 pillars on ChatGPT, Claude,
> Perplexity, Gemini, and Google AI Overviews, then ship the fix.

## Key Pages
- [Homepage](https://example.com/): Hero, product overview, live score preview
- [Pricing](https://example.com/pricing): Plans and what's included at each tier
- [Docs](https://example.com/docs): Setup and API reference

## Use Cases
Solo SaaS founders who see competitors named by ChatGPT and Perplexity
on buyer-intent queries and want to close that gap without hiring a
marketing team or briefing an engineer for every fix.

Notice what's missing: no keyword stuffing, no repeated brand name for emphasis, no boilerplate. A model doesn't need convincing, it needs facts it can retrieve accurately.

How to Generate One Without Writing It by Hand

Writing a good llms.txt from a blank page means deciding which pages matter, summarizing each one accurately, and phrasing the use-cases section the way a model would want to read it, which is exactly the kind of small-but-fiddly task that's easy to keep deferring. GetIntel's llms.txt generator drafts one from your actual site content and product description rather than a generic template, which is the difference between a file that describes your product and one that describes "a SaaS tool."

Once it's drafted, the fastest way to ship it is the same pattern covered in our guide to shipping AI-visibility fixes without an engineering backlog: hand the file content to Claude Code or Cursor with instructions to create it at your domain root and open a PR. It's a single new file, low-risk to review, and it deploys on your normal release cadence.

How to Know If It's Actually Improving Your Citation Rate

This is the step almost everyone skips. Shipping the file isn't the finish line, confirming it changed something is.

1. Before you ship, record your baseline. Pick the 5-10 buyer prompts where you're currently missing, and note exactly how each engine answers them today.
2. Ship the llms.txt file through your coding agent, and confirm it's live and returns a 200 at yourdomain.com/llms.txt.
3. Wait for a real recrawl window. AI crawlers don't refresh instantly. Give it at least one to two weeks before checking again, longer if your domain is new or has a thin crawl history.
4. Re-run the exact same prompts. Not similar prompts, the same ones, so you're comparing like for like.
5. Track the trend, not a single check. One re-probe can be noisy, since AI answers vary run to run. A tool that tracks your AI visibility score daily turns this into a trend line instead of a single data point you have to interpret.

If the citation rate on those specific prompts hasn't moved after a few weeks, llms.txt alone probably isn't the bottleneck, the more likely gap is third-party authority (backlinks, Reddit mentions, review sites), which llms.txt can't manufacture on its own.

Common Mistakes

Treating it as a keyword-stuffing opportunity. Repeating your brand name or target keywords throughout the file doesn't help; it just makes the file less useful as an accurate summary.

Listing every page on the site. The value is in curation. A model that has to sift through 150 URLs to find the three that matter gets less out of the file than one that gets ten well-described pages.

Writing it once and never updating it. If your key pages or positioning change and the file doesn't, you're actively feeding a model an outdated summary of your own product.

Never checking whether it worked. A file that's live but never re-verified against real buyer prompts is a fix you're assuming worked, not one you've confirmed.

Reference Table

---

Want a real llms.txt drafted from your actual site instead of a template? Generate yours free and see what a curated version of your own content looks like.