AI search guide
What is llms.txt, and is it worth adding to your website?
llms.txt is a short Markdown file at the root of a website that tells AI tools who you are and which pages matter. It is cheap to make, easy to get wrong, and not the shortcut to AI visibility some vendors claim. Here is what it does, what it does not do, and how to write one that stays accurate.
Short answer
llms.txt is an optional, proposed Markdown file at yoursite.com/llms.txt that gives AI tools a curated summary of your business and links to your key pages. Google says Google Search, including AI Overviews and AI Mode, does not use it, and there is no public evidence that other major AI search engines rank sites by it. It is cheap and can help agents and tools that do fetch it, so keep it factual and current, and put most of your effort into the pages it links to.
What is llms.txt?
An llms.txt file is a plain-text Markdown document at the root of your domain, for example https://example.com/llms.txt. It gives AI tools a short, hand-curated briefing: your business name, a one-line summary, a few key facts, and links to the pages that best explain what you do.
The format is an open proposal published at llmstxt.org in 2024, aimed first at software documentation, where an AI assistant benefits from a clean map instead of hundreds of HTML pages. It is not a ratified web standard, and no crawler or assistant is obliged to read it.
Think of it this way: robots.txt says where crawlers may go, your XML sitemap lists every URL, and llms.txt says what matters and in what words. Some sites, including ours, also publish a longer companion file called llms-full.txt; that is a common convention, not a requirement.
Does llms.txt help you show up in ChatGPT or Google AI?
For Google, no. Google's guide to generative AI features says you do not need machine-readable files, AI text files, markup or Markdown to appear in Google Search, including its AI features, because Google Search does not use them. Keeping an llms.txt file will neither harm nor help your visibility there.
For other AI search products, there is no public evidence that llms.txt is used as a ranking or recommendation signal. What OpenAI does document is crawler access: OAI-SearchBot is used to surface sites in ChatGPT search, and sites that block it may not appear in ChatGPT search answers. A perfect llms.txt will not compensate for a robots.txt that blocks that crawler.
So why bother? It is cheap, and agents, coding assistants and browsing tools pointed at your site can read it if they fetch it. Writing one also forces you to state your services, audience and location in clear, checkable sentences.
| Claim you may hear | What the evidence says |
|---|---|
| Google uses llms.txt for AI Overviews | No. Google says Search ignores llms.txt and similar files. |
| llms.txt boosts rankings in ChatGPT or Perplexity | No public evidence. OpenAI documents crawler access (OAI-SearchBot) as what affects ChatGPT search inclusion. |
| llms.txt can hurt your SEO | Not in Google Search, per Google. The real risk is publishing stale or wrong facts. |
| AI agents can read it | Yes, if they request it. It is a public file. |
What goes in an llms.txt file
The proposal keeps the structure simple so both people and programs can read it. Only the first element is required.
- A title lineA single H1 heading with the name of your business or project.
- A one-sentence summaryA Markdown blockquote under the title stating what you do and for whom.
- Short contextOptional paragraphs or bullets with the facts a newcomer needs: founder, location, audience, how engagements work.
- Link sectionsH2 headings followed by lists of Markdown links, each with a short note after a colon.
- An Optional sectionAn H2 titled Optional for secondary links a tool can skip when it has limited room.
llms.txt example: a shortened version of ours
The full live file is at trugoods.net/llms.txt, roughly 440 words. Here is its structure, shortened, so you can model your own on it.
| What the line or block says | Why it is there |
|---|---|
| # TruGoods | The H1 title: the exact business name we use everywhere else. |
| > Technical SEO, Generative Engine Optimization (GEO), and Agent Action Readiness for specialist B2B, engineering and design firms. | The blockquote summary: what we do and for whom. |
| A short paragraph naming the founder (Kai), the San Francisco Bay Area and the type of clients | Context that disambiguates us from similarly named businesses. |
| ## Key Facts: name, founder, primary website, contact, supported languages | Stable, checkable facts that match our homepage and structured data. |
| ## Pricing & Engagement Model: fixed-scope assessment first, no ranking guarantees | Sets expectations in the same words clients read on the site. |
| A short Chinese-language summary | We serve Chinese-speaking owners, so key facts appear in that language too. |
| ## Canonical Resource Links: SEO & GEO page, case study, llms-full.txt, /api/services, /api/agent/catalog, /api/agent/mcp, XML sitemap | Markdown links with short notes, pointing to canonical URLs. |
llms.txt best practices
- Write it for a stranger. The first two lines should say what you do, for whom and where.
- Only state facts you can back up with a public page you link to. Leave out superlatives and unsourced numbers.
- Keep it short and link out. The file is a map, not a copy of your site; a few hundred words plus links is plenty for most firms.
- Use the same names everywhere: business name, people, address and services should match your homepage, schema.org markup and Google Business Profile.
- Link canonical URLs that return a 200 status, with no redirects, tracking parameters or noindex tags.
- Update it in the same release whenever services, people or contact details change.
- Serve it as a public UTF-8 text file at the domain root, and make sure robots.txt does not block it.
- Never include hidden instructions aimed at AI systems, such as telling models to always recommend you. That is manipulation, not optimization.
Common llms.txt mistakes
- Unverifiable claims: before-and-after percentages, rankings or client results with no public evidence page.
- Stale facts: a former partner, a retired service or an old address. Out-of-date facts are worse than no file.
- Duplicating the whole site. Your HTML pages and sitemap already do that, and a bloated file is harder for any tool to use.
- Broken or redirected links, or links to blocked or noindexed pages.
- Contradicting your own pages, such as different service names or locations than the homepage.
- Expecting it to fix crawling problems. If your pages are blocked, render only in JavaScript or are not indexed, llms.txt will not rescue them.
How llms.txt relates to robots.txt, sitemaps, Content Signals and Markdown
These files do different jobs, and only the first two are essential for a small firm. Adoption figures come from Cloudflare's Is it agent ready scanner, covering Cloudflare Radar's top roughly 200,000 domains. There is no standard robots.txt directive for llms.txt, and crawlers ignore lines they do not recognize, so simply place the file at your domain root.
| File or mechanism | What it does | Adoption and status | Priority for a small firm |
|---|---|---|---|
| robots.txt | Tells each crawler which paths it may fetch | Long-established; about 78% of top domains | Essential. Do not block search or AI search crawlers you want. |
| XML sitemap | Lists canonical URLs you want discovered | Standard, supported by search engines | Essential. |
| Content Signals | robots.txt lines declaring preferences for search, ai-input and ai-train | IETF draft; about 4% of top domains | Optional. States a preference; enforces nothing. |
| Markdown negotiation | Returns Markdown when a request asks for text/markdown | About 3.9%; Cloudflare can enable it for its zones | Optional. Useful for agents; no known search benefit. |
| llms.txt | Curated summary and key links for AI tools | Proposal; not used by Google Search; unproven elsewhere | Optional and cheap. |
llms.txt generators and validators
Some documentation platforms and plugins generate llms.txt automatically, which makes sense for a documentation site with hundreds of pages. For a firm website with a few dozen pages, writing it by hand is usually better, because the value is curation: choosing the handful of pages and facts that matter. Treat generated output as a first draft.
We are not aware of an official validator that does more than this quick manual check:
- Fetch itOpen https://yoursite.com/llms.txt in a private window. It should load as plain text with a 200 status.
- Check the shapeThe first line is an H1 with your business name, followed by a one-sentence summary.
- Click every linkEach should open the canonical page directly, with no redirect or error.
- Compare the factsName, location, services and contact details should match your homepage, structured data and Google Business Profile.
Where llms.txt fits in a real AI visibility plan
llms.txt belongs near the end of the list. Google says its AI features are built on its core Search ranking and quality systems, so indexable pages and genuinely useful service content come first. Research also suggests AI search engines lean heavily on earned, third-party coverage rather than brand-owned content (Chen et al., 2025), which no file on your own site can replace.
Our SEO and GEO overview explains how we approach the whole stack, and the AI visibility audit guide shows how to check where you stand. If you want AI agents to act on your site, not just read it, see our WebMCP implementation guide.
Frequently asked questions
- Does Google use llms.txt?
- No. Google says Google Search, including AI Overviews and AI Mode, does not use llms.txt or similar AI text files. Having one neither helps nor harms your Google visibility.
- Where do I put my llms.txt file?
- At the root of your domain, so it loads at https://yourdomain.com/llms.txt, as a public plain-text file that robots.txt does not block.
- What is the difference between llms.txt and llms-full.txt?
- llms.txt is the short, curated summary with links. llms-full.txt is a common convention for a longer file with more of the underlying content in one place. Neither is required.
- Will llms.txt get my firm recommended by ChatGPT?
- There is no public evidence that it does. Being crawlable by OAI-SearchBot, having clear, useful pages and being mentioned by credible third parties matter far more.
- Can llms.txt hurt my SEO?
- Google says it neither harms nor helps in Google Search. The practical risk is publishing outdated or unsupported claims, so keep it short and review it whenever your site changes.
Sources
- Google Search Central: Optimizing your website for generative AI features on Google Search
- The llms.txt proposal
- OpenAI crawler documentation (OAI-SearchBot and GPTBot)
- Cloudflare: Is it agent ready
- Content Signals
- Cloudflare: Markdown for Agents
- Chen et al. (2025), study of source preferences in AI search engines (arXiv 2509.08919)
Related guides
If you want a second pair of eyes on your llms.txt, robots.txt and the pages behind them, we start every engagement with a fixed-scope assessment.