Should your website have an llms.txt file? An honest guide for 2026

There’s a new text file popping up on developer and SaaS sites. It’s called llms.txt, and it’s supposed to help AI figure out which parts of your site actually matter.

The file is a proposed standard for guiding large language models to your best content. It’s a bit like robots.txt for AI, but right now, no big AI company has confirmed they use these files when they crawl sites. If you’re just curious or experimenting, maybe give it a go, but don’t expect instant results.

We’ve tried the file ourselves, watched how crawlers behave, and looked at the evidence. Here’s what actually happens when you add llms.txt to your site, who’s using it, and whether it’s worth your time in 2026.

How llms.txt works and what sets it apart

The llms.txt file sits in your site’s root folder. It’s just plain text that tells AI models which pages you care about most.

You write it in markdown, so it’s a list of URLs with short descriptions. This makes it easier for large language models to parse your content, compared to digging through normal HTML pages.

What goes in an llms.txt file

It’s a hand-picked list of your top URLs, each with a short description. You use markdown—so # for headings, - for bullet points, and links like text.

Usually, you start with your company name and a quick summary of what you do. Then you break things into sections, maybe “Services”, “Docs”, or “Case Studies”. Each URL gets a one-liner explaining what’s on that page.

Some folks add extra bits. BX3 Interactive includes target terms and calls to action for each URL. Hugging Face puts installation commands and code samples right in their llms.txt, so AI doesn’t have to dig through endless docs.

You put the file at yoursite.com/llms.txt. If you only want to cover specific stuff, you can drop it in a subdirectory, like docs.yoursite.com/llms.txt.

How llms.txt differs from robots.txt and sitemaps

Robots.txt is about access. It tells crawlers what they can and can’t reach using “Allow” and “Disallow”.

An llms.txt file doesn’t block or allow anything. It just points AI models to your best pages and suggests what’s worth their attention.

XML sitemaps list all your URLs for search engines. They’re meant to be comprehensive. llms.txt is more selective, focusing on the pages you want AI systems to prioritise and cite.

You need both. If robots.txt blocks an AI crawler, llms.txt won’t help because the bot can’t reach your pages. If you only have robots.txt, AI crawlers can get in, but they don’t know what’s important.

Why markdown matters for llms.txt

Markdown keeps the file readable for both people and AI. You use # for main headings, ## for sections, and ### for subsections. Links look like anchor text, and you can add a description after with a colon.

This structure helps because AI models parse markdown more smoothly than HTML cluttered with nav bars, scripts, or ads. The file gives models clean, structured info without all the noise.

The hierarchy also matters. Top-level headings show broad categories, and nested ones show how pages relate. AI can map out your site’s structure quickly.

Adoption and evidence: who’s actually using llms.txt

In 2025, 86% of websites had an explicit AI training policy. Most went for selective permissions. Big AI names like OpenAI, Anthropic, and Google have adopted the standard, and leading AI crawlers follow these rules over 89% of the time.

Adoption rates across different sites

One study of 2,147 sites in 15 industries found that 68% allow some form of AI training. Only 18% block it completely, which is less than you’d think.

About 45% use selective policies, usually allowing blog posts and docs while blocking admin pages and user data.

Industries handle this differently. Education is the most open, with 42% allowing all AI training. Healthcare is strictest at 51% blocking everything, mostly for privacy reasons. E-commerce prefers selective rules—61% allow product catalogues but protect customer info and reviews.

Tech and government site examples

Tech leads the way—95% have clear policies. Most let AI access docs but keep code and customer data private.

Companies like Cloudflare, Anthropic, Vercel, and Supabase are early adopters. Mintlify and Ahrefs have the file on their docs, too.

“Disallow: /admin/” is everywhere—89% of sites with selective rules use it. “Allow: /blog/” shows up in 73% of selective cases. Tech companies use “Allow: /docs/” in 61% of cases.

How major AI providers use llms.txt

OpenAI’s GPTBot follows llms.txt rules 94% of the time. Anthropic’s Claude-Web is at 91%. Google-Extended, which runs Gemini, follows them 89% of the time.

These stats come from tracking over 50,000 AI crawler requests. High compliance means ChatGPT, Claude, and Gemini usually respect what you put in your llms.txt.

Nearly half of sites with selective policies set different rules for different AI bots. GPTBot gets mentioned in 68% of these, Google-Extended in 52%, and Claude-Web in 28%. Perplexity and others don’t get as much attention yet.

Impacts on AI search, SEO, and visibility

An llms.txt file won’t replace your usual SEO. It just sits alongside it and gives AI a clearer idea of what your site does and which pages to pull from when generating answers or citations.

How llms.txt shapes AI-generated answers

AI assistants like ChatGPT, Perplexity, and Claude handle millions of prompts daily. When someone asks a question your site could answer, these tools pick pages to reference based on how well they understand your content.

llms.txt gives them a tidy summary of your top pages. It spells out what’s on each one and why it matters.

If you don’t have one, AI still crawls your site. It just has to guess what’s relevant, which means it could miss important pages, misinterpret your business, or skip your site if things aren’t clear.

Visibility in AI search matters because visitors from AI answers are usually more qualified than typical organic traffic. Even a small bump in AI citations can mean more of the right kind of visitors.

Citations and brand mentions

When an AI tool cites your site, it usually links to a page and names your brand. That depends on whether the AI understands your business and trusts your content.

llms.txt boosts your odds of accurate citations. It brings the right pages to the surface and gives context to help AI describe your brand properly.

We saw this with Triangle IP, a patent management tool. Their llms.txt highlights conversion-focused content. When patent lawyers or IP teams ask AI about workflows, those pages show up in the answers. The same blog posts helped Triangle IP get 500+ signups.

Citations won’t always bring traffic, but they do put your brand in front of people who are already interested.

Does llms.txt affect rankings?

llms.txt doesn’t change your Google search rankings. It’s separate from things like page speed, schema, or backlinks.

What it does change is how often AI tools reference your site in answers. That’s more about being visible than about classic rankings. Traditional SEO still matters for how you show up in large language models, but llms.txt lets you steer which pages get the spotlight.

AI overviews in Google search already pull from indexed content. llms.txt won’t change that directly, but it might affect how other AI assistants treat your site when they build responses.

It’s more about discoverability than a straight ranking boost.

How llms.txt fits with technical SEO and sitemaps

Your sitemap.xml tells Google which pages you have. Robots.txt says what crawlers can access. llms.txt tells AI which pages matter and what they’re about.

All three work together. You still need clean HTML, fast load times, and proper metadata. You also need to let AI crawlers in using robots.txt.

llms.txt adds an AI-specific layer. It doesn’t replace sitemaps or schema. It just gives AI tools a curated view of your content so they don’t have to wade through everything.

If you run a blog-heavy site or a SaaS with detailed docs, this can really matter. It focuses AI on the pages that show off your expertise or nudge people to sign up, instead of leaving them to guess.

Implementation, costs, and best practice right now

Making an llms.txt file costs nothing. If your site is simple, you can get it done in about 20 minutes.

The format is free, open, and doesn’t need any paid tools or ongoing fees.

How to create and update an llms.txt file

You write an llms.txt file in plain markdown and save it as llms.txt in your site's root directory. That's the same spot as robots.txt.

The specification asks for one thing: an H1 with your site or project name. After that, add a blockquote summary, any detail you want, and H2 sections with markdown lists that link to your key pages.

Each link can have a colon and a short note about what the page covers. It's all pretty open, so you can keep it as simple or detailed as you like.

Jeremy Howard, who proposed the standard, suggests making markdown versions of important pages at the same URL with .md added. This works well for documentation sites where you already have clean content.

If your pages are buried in messy HTML, this bit takes longer. It's a pain, honestly, but worth it if you want things tidy.

You can check your file with tools like llmstxtvalidator.dev before publishing. Yoast hasn't added llms.txt support yet, so you end up uploading or editing by hand via FTP or your CMS.

We look at server logs every few weeks to see if LLMs request the file. So far, traffic is tiny as of April 2026.

Integration with developer documentation and API references

Mintlify and other docs platforms now add llms.txt generation features that spit out the file automatically from your docs structure.

Developer sites get the most out of this since LLMs need fast access to API references, code snippets, and library docs.

If you run a SaaS with public docs, link to your getting started guide, API endpoints, and auth pages under a "Docs" H2. You can add an "Optional" section for less crucial stuff like changelogs or advanced config, which LLMs usually skip if context is tight.

We set this up on client docs sites where devs ask AI tools for integration help. The file gives models a map, so they don't have to scrape your whole site.

Pricing doesn't come into it, since you're just organising links you already have.

Working within current limitations and alternatives

When is it worth spending time maintaining llms.txt, and when can you leave it alone?

If you already keep your docs, API references, or product lists in markdown, making an llms.txt file is quick. It usually takes just a few minutes.

There's no real harm in adding one, though you probably won't see any obvious gains.

Keep maintaining llms.txt only if your content changes a lot and you think AI providers might actually use the standard. Otherwise, just set it up once and move on.

If you don't use markdown or your site is simple enough for AI crawlers to handle, you can skip llms.txt. You're better off spending time on things like fixing crawl errors or making your site faster.

So far, we've never seen llms.txt help with AI retrieval or drive more traffic. Maybe it'll matter one day, but honestly, it could just disappear like plenty of other web standards that went nowhere.