No. An llms.txt buys a Swiss SME no measurable visibility in AI answers today. It does no harm either, and it takes minutes to write. No major AI provider has confirmed that it reads other people’s llms.txt files at answer time, and Google states in its own documentation that Search ignores such files.
At Carigiet GEO we hear this question regularly, and it almost always comes from outside: from a technical GEO audit that lists the missing file as a defect and tells the company to create one. The number behind it is uncomfortable. Of roughly 38,000 websites with a valid llms.txt, 97% received not a single request for that file in May 2026, from bots or from humans. That comes from an analysis of 137,210 domains (Ahrefs, June 15, 2026).
For Generative Engine Optimization, that makes the file a footnote.
In short
- The answer: No, for a typical Swiss SME an llms.txt is not a visibility lever. It does no harm either.
- The evidence: 97% of published files were never fetched in May 2026. A second study across roughly 300,000 domains found no statistical link between llms.txt and AI citations.
- The Google finding, in two parts: Google Search does not use these files, by its own documentation. Google’s Lighthouse still checks whether an existing llms.txt follows the recommendation. If the file is missing entirely, the audit counts as not applicable.
- The most common error: An llms.txt does not steer crawlers. It allows nothing, blocks nothing, and holds no access rules.
- The exception: Public developer and product documentation that AI coding tools read. There the benefit is documented.
- What counts instead: Do the AI crawlers reach your site, is your content in the delivered HTML, is the technical base clean, and are you named on the sources that get cited for your topic?
- Our own position: carigiet.com carries an llms.txt, although we sell it to nobody as a service. Why, is written out below.
What is an llms.txt?
An llms.txt is a Markdown file in the root directory of a website that lists its most important pages with one short description each. It sits at example.com/llms.txt and is meant for language models, not for visitors.
Jeremy Howard, co-founder of Answer.AI and fast.ai, proposed it on September 3, 2024. The original proposal puts it literally: “We propose adding a /llms.txt file to websites that are designed for reading by language models, not just humans.” The reasoning: the context windows of language models are too small for whole websites, and HTML with navigation, advertising and JavaScript converts into clean text only imprecisely.
What matters for the decision is the status. The specification is to this day an informal proposal by a single person, last updated on August 10, 2026. There is no RFC, no working group at the IETF or W3C, and no ratification by a standards body. The structure is plain: one H1 with the name of the website as the only mandatory part, below it a short summary and sections with link lists.
You can see what that looks like in our own file: carigiet.com/llms.txt (in German; the structure is the point). In engagements we almost always meet the same shape when a file already exists: a sitemap with one line of description per page.
Alongside it sits a second, less widely implemented file: the llms-full.txt. It embeds the full text of the linked documents into a single Markdown file, so that a model can load the entire content into its context window at once. The llms.txt is the short index, the llms-full.txt the complete body of text.
llms.txt, robots.txt, sitemap.xml and structured data: what does what?
Of these four files, only the robots.txt controls access. The llms.txt controls nothing.
| File | What it contains | Who demonstrably reads it | Does it control crawler access? |
|---|---|---|---|
| llms.txt | Curated Markdown overview of the most important pages with descriptions | No major AI provider confirms that it reads it; occasionally AI coding tools and humans | No |
| robots.txt | Rules for which areas a crawler may visit (RFC 9309, 2022) | All major AI crawlers; compliance is voluntary | Yes |
| sitemap.xml | Machine-readable list of the indexable URLs with their change dates | Search engine crawlers | No |
| Structured data (Schema.org, JSON-LD) | Machine-readable description of what is on a page | Search engines, established for rich results | No |
This distinction is regularly misrepresented in SEO content and in AI-visibility audits. A common claim is that an llms.txt lets you steer AI crawlers, sometimes with invented directives such as “Allow”, “Disallow”, “Crawl-Delay” or “NoTrain”. Not one of them appears in the specification. The file is a table of contents, not a rulebook.
Anyone who actually wants to govern access does it in two places: in the robots.txt and at the server or WAF level. The second one is the place nobody in an SME usually knows about. Blocks at Cloudflare, Akamai or an AWS WAF, harmless rate limits included, can shut AI crawlers out without any of it ever appearing in the robots.txt.
What do the data say about the effect of an llms.txt?
The two largest independent studies reach the same result: no measurable effect.
The requests. Ahrefs checked all 137,210 domains that had traffic in its own Web Analytics in May 2026 for a valid llms.txt, then evaluated every request for that file by status and user agent. 28% of the domains published a file. Of those, 97% received not a single request in the entire month. Of the 3% with any requests at all, only 1.1% of all requests came from AI retrieval bots such as OAI-SearchBot or PerplexityBot. The list was led by SEO audit tools at 21.7%. For domains without an llms.txt, the share of AI bots that asked for it anyway was zero.
The citations. SE Ranking tested roughly 300,000 domains with correlation tests and a predictive model to see whether the presence of an llms.txt relates to how often a site is cited by AI. The result: no link between llms.txt and AI citations (SE Ranking, November 7, 2025). Removing the llms.txt variable even improved the accuracy of the predictive model.
The adoption. It is growing fast, and that is the strongest counter-argument. A tracker across more than three million websites counted around 4,100 files in June 2025 and around 36,100 in May 2026, an 8.8-fold rise in twelve months. The study’s own conclusion is sober all the same: more and more websites carry the file, and almost nothing reads it.
Both large studies come from SEO tool vendors, who in a broad sense are interested parties. We use them because sample, period and method are documented, and because they confirm each other independently. Ahrefs explicitly calls its own numbers an upper bound, because a request does not prove that a model read or used the content.
This is exactly the point at which the case for the file collapses. The British search consultant Mark Williams-Cook published a joke standard called cats.txt on August 4, 2026, a file for declaring the office cats, complete with a “PurrLevel” metric scored out of ten. It passed the same four “proofs” used to argue that llms.txt works: AI bots crawled it, Google indexed it, language models reproduced details about a cat that exists only in that file, and ChatGPT confirmed at length that cats.txt could help with ranking. A crawler request proves that the crawler works. Nothing more. From 25 years of software development we know the difference between a file a server delivers and a file a system actually evaluates. For the llms.txt, the second part is unproven to this day.
Which AI providers read an llms.txt?
No major provider has confirmed that it reads other people’s llms.txt files at answer time. Google says no explicitly.
| Provider | Official statement on llms.txt | How access is actually governed |
|---|---|---|
| Explicit no in the documentation | robots.txt (Googlebot, Google-Extended) | |
| OpenAI | No evaluation documented; the crawler docs name robots.txt only | robots.txt (GPTBot, OAI-SearchBot, ChatGPT-User) |
| Anthropic | No recommendation to site owners; not mentioned in the crawler article | robots.txt (ClaudeBot, Claude-User, Claude-SearchBot) |
| Perplexity | No evaluation confirmed; site owners are pointed to robots.txt | robots.txt (PerplexityBot) |
| Microsoft Bing | No public statement findable | robots.txt (Bingbot) |
| Mistral | No public statement | robots.txt |
Google is the only provider with a documented no. Its AI optimization guide (Google Search Central, last updated July 10, 2026) says literally, in the section on common misconceptions: “You don’t need to create new machine readable files, AI text files, markup, or Markdown to appear in Google Search (including its generative AI capabilities), as Google Search itself doesn’t use them.” And further: such files neither harm nor help visibility, because Google Search ignores them.
There is a named statement on top of that. John Mueller, Search Advocate at Google, wrote on June 17, 2025: “FWIW no AI system currently uses llms.txt”.
Why Google’s own tool checks the file anyway
Anyone who now opens PageSpeed Insights finds what looks like the opposite. The tool runs Lighthouse and shows a fifth category next to Performance, Accessibility, Best Practices and SEO: “Agentic Browsing”. One of its three audits is called “llms.txt follows recommendations”. On carigiet.com it shows 3 of 3 audits passed (checked on August 23, 2026).
That is not a yes to the effect, and Google says why itself. In the interface the category carries the note that it is “still under development and subject to change”. The documentation for the audit (Chrome for Developers, as of May 5, 2026) records what happens when no file is present: “If the file is not provided by the server (resulting in a 404), the audit is marked as Not Applicable (N/A), as providing the file is optional at the moment.” What gets checked, then, is the form of a file that exists. The audit itself asks for a Markdown file with at least one H1. If you carry no file, you lose nothing there.
The reasoning inside the audit is worth noting. Without a correct file, it says, language models cannot understand “how you want your website to be crawled or used for training”. That is precisely the attribution corrected above: the llms.txt governs neither crawling nor training. It makes plain where the common reading comes from. It is written into the very tool many companies open first.
At OpenAI, Anthropic and Perplexity the situation is identical and still often misread: all three publish an llms.txt for their own documentation, and from that people conclude that they evaluate other people’s files. Their crawler documentation says otherwise. OpenAI governs GPTBot, OAI-SearchBot and ChatGPT-User through robots.txt, Anthropic does the same for its three bots, and Perplexity points site owners to the same file. Not one of the three documents says anything about evaluating other sites’ llms.txt files. Where “llms.txt” appears there at all, it is the link to their own documentation index. That is exactly where the confusion comes from.
What would overturn this no can be named precisely: a confirmation in the official documentation of a major provider, a clear rise in requests from AI retrieval bots, or a solid study showing a link between llms.txt and citations. If one of those happens, we change our recommendation. Until then, today’s evidence stands.
When is an llms.txt worth it after all?
In exactly one case the benefit is documented: for public developer, API or product documentation that AI coding tools read.
In the Ahrefs data, Claude Code, Anthropic’s coding agent, was the second-largest named AI requester of llms.txt files after GPTBot, ahead of every AI search and assistant bot. The reason is mundane: these tools fetch the file because a user points them at it or because it is linked in the documentation, not because they go looking for it speculatively. The effect was amplified when the documentation platform Mintlify rolled the file out automatically across all hosted docs pages in November 2024.
The second case is simpler still. If your system generates the file anyway, leave it there. Part of the high adoption figures goes back to platforms that ship the file by default, without anyone ever having decided to. In that case, just check that no outdated, internal or dead URLs appear in it.
For a Swiss B2B SME without public developer documentation, the documented use case does not apply. The exception is software and technology companies whose customers work with coding agents.
The counter-check matters just as much. An llms.txt demonstrably does not do the following:
- It gives no access when your crawlers are blocked at server or CDN level.
- It does not replace content that has to be in the delivered HTML.
- It does not influence Google Search, positively or negatively.
- It stops no model from using your content for training.
- It does not produce a mention in an AI answer.
Why carigiet.com carries an llms.txt anyway
Because it cost us an hour, displaces nothing, and is run here as an experiment, not as a service. Our own technical audit explicitly recommends spending no effort on it, and the file sits at carigiet.com/llms.txt all the same. We would rather write that contradiction out than route around it.
Four reasons sit behind it. First, we wanted to see for ourselves what actually gets fetched, instead of taking it from studies. Second, we wanted to know how such a file is built properly, in case it counts one day. Third, audit tools reward a correctly built file with a passed audit, Google’s Lighthouse included. We wanted to know what exactly gets checked there. What that third reason is worth stands above: if you carry no file, you lose nothing there. Fourth, we consider it possible that an assistant starts using the file once enough websites carry one. We do not run an llms-full.txt.
No recommendation follows from that. The rule here: a measure that does not measurably work is not something we sell as a service. A cheap, harmless experiment may run as long as it is declared as one and displaces nothing that works. The llms.txt is a fixed audit point in our technical audit, the finding is documented, and where there is no need to act, none is manufactured. How we work in general is set out in our published working principles.
From our engagements comes the same observation: on client sites that already carried an llms.txt, nothing changed in the measurement. That is an observation, not a study result. The large studies measure requests and citations, not the visibility values of individual engagements. Both point in the same direction, and neither replaces the other.
What actually decides whether AI systems read and cite you
When somebody asks us about the llms.txt in a first call, our answer is: do assistants look at it, clearly no, not yet. We would far rather concentrate on a clean technical foundation and a tidy Google Search Console, no 404 errors, every important page indexed. That sounds less exciting than a new file and decides more.
These five points we check, in this order:
- Do the AI crawlers reach your site at all? That is the question before all others. How to check: look in the robots.txt to see whether GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot are allowed, then check in the server logs or in the WAF configuration whether their requests are turned away before they ever reach the website.
- Is your content in the delivered HTML? The great majority of AI crawlers run no JavaScript. A joint analysis by Vercel and MERJ across more than 500 million GPTBot requests (December 17, 2024) records it literally: “none of the major AI crawlers currently render JavaScript”. The ChatGPT crawlers did load JavaScript files in 11.5% of requests, but did not execute them. Purely client-side rendered text is practically invisible to ChatGPT, Claude and Perplexity, even when it performs well in Google Search. How to check: open the page as source code in the browser and see whether the body text is there.
- Is the technical base clean? No 404 errors on important pages, every core page indexed, redirects without chains. How to check: work through the page indexing report in Google Search Console and resolve every non-indexed core page one by one.
- Is structured data present? Schema.org markup is useful infrastructure, but not a confirmed lever for AI citations. Ahrefs tracked 1,885 pages that added JSON-LD against 4,000 control pages (Ahrefs, May 11, 2026) and found no major uplift in citations on any platform. For classic rich results it remains worthwhile. How to check: send one core page through the Rich Results Test.
- Do you appear on the sources that get cited for your topic? Anyone who does not appear in those sources does not exist in the answer. This is where the largest lever and the most work sit. The most-cited research paper on the subject (Aggarwal et al., KDD 2024) puts the effect of editorial preparation, meaning quotations, source citations and statistics in the text, at “up to 40% in generative engine responses”. That is an upper bound from a simulation, not an expected average. How to check: ask the same buyer question several times and note which sources the answer cites.
We have written up the self-test for this in a guide with ready-made prompts: how to check your AI visibility yourself. What that concretely involves on your own website is here: what ChatGPT optimization covers.
Expect a mention from no single measure, not from an llms.txt and not from anything else. AI answers are statistical, and the same question does not return the same result twice. Why there is no ranking in ChatGPT in the classical sense we have written up separately. And whether a measure works is decided in the end by the measurement, not by opinion: how AI visibility can be measured reliably.
The decision rule: when yes, when no
You can forward this block without sending the rest of the article with it.
Yes, if you run public developer, API or product documentation. Publish the file for that documentation, link it in the HTML, keep it current and treat it like code: version control, restricted write access, change notifications, and only your own resources linked from it.
Yes, if your platform generates the file anyway. Leave it there. Check once that no internal, outdated or dead URLs are in it, and then forget it.
No, if you are asked to pay for it. The basic job takes minutes to do yourself, or the platform generates it.
No, if the file takes a place on your priority list. Whoever has invested the hour has lost nothing. Whoever postpones the work that works because of it, loses.
And if you want to know for yourself: the only effort that pays off in our view is a comparison of your own. Take two comparable pages, give one of them an entry in the llms.txt and the other none, and measure over several weeks which one appears more often in AI answers. In parallel, filter your server logs for /llms.txt and for AI user agents. If you see not a single request from an AI retrieval bot over a month, the question is answered for your company.
In both cases, yes and no, the next action is the same: check first whether the AI crawlers reach your website and whether your content is in the delivered HTML.
Frequently asked questions about llms.txt
What is the difference between llms.txt and robots.txt?
The robots.txt governs access, the llms.txt describes content. The robots.txt is an established standard (RFC 9309) that serious crawlers follow, and it governs which areas may be visited. The llms.txt is an informal proposal with no access rules: it allows nothing, blocks nothing and throttles nothing. The similar file name is the cause of the confusion.
Does ChatGPT read my llms.txt?
There is no evidence that it does. OpenAI’s crawler documentation describes GPTBot, OAI-SearchBot and ChatGPT-User purely through robots.txt and says nothing about evaluating other sites’ llms.txt files. The confusion comes from OpenAI publishing an llms.txt for its own documentation, which says nothing about what its crawlers read elsewhere. And a confident yes from ChatGPT itself is not evidence: in the cats.txt experiment above, ChatGPT confirmed at length that a joke file about office cats could help with ranking.
My PageSpeed report lists llms.txt. Do I need to act?
No, as long as it shows a passed audit or “Not Applicable”. Google’s PageSpeed Insights checks in the “Agentic Browsing” category whether an existing llms.txt follows the recommendation, meaning that it is a Markdown file with at least one H1. If the file is missing entirely, the report counts the point as not applicable, per Google’s documentation, because the file is “optional at the moment”. You only need to act if you carry a file and the audit fails. In that case the form is broken, and the fix takes minutes.
Can an llms.txt hurt?
No, as long as it is maintained correctly. Google states explicitly that such files neither harm nor help visibility in Search. Two points are worth the attention all the same: do not link internal or confidential URLs, and do not write instruction-like sentences into it that a model could read as a command.
Do I need an llms-full.txt as well?
No, not for the vast majority of websites. The llms-full.txt embeds the entire text of all linked documents into a single file and is meant for large documentation sets that a model should load in one piece. For a company website with a few dozen pages it adds nothing the normal website does not already deliver. We do not run one ourselves.
How do I check whether my llms.txt has ever been fetched?
Filter your server logs or your bot analytics tool for requests whose path contains llms.txt, and look at the user agents: GPTBot, OAI-SearchBot, ClaudeBot, Claude-User and PerplexityBot are the relevant ones. Three things distort the picture. A request does not prove that the content was read. Blocks at CDN or WAF level can turn crawlers away before they ever appear in the log. And link preview bots from messaging apps fetch the file too, without any AI system being involved.
Should I pay to have an llms.txt created?
No. The file takes minutes to produce, or the platform generates it automatically. We regularly see it listed as a defect in technical GEO audits, together with the demand to create one. That makes a serviceable test of the provider behind it: anyone who writes an llms.txt into a proposal as a visibility measure should be able to say which provider evaluates it. Which other questions help in a first call is set out here: eight checks for choosing a provider.
Where you stand today
The llms.txt is a small question. The large one behind it is this: are you named at all when somebody asks an AI system for a provider of your kind, and which sources does the answer cite while doing it?
That question can be answered, with numbers rather than assumptions. Our free AI Visibility Analysis shows you where you stand today across the common AI systems. If you would rather walk through the results with somebody, book a 15-minute first call. No sales pitch, no slides, and at the end you will know whether the effort is worth it for you.
Related posts
Check Your AI Visibility Yourself: Test in 30 Minutes Whether ChatGPT Recommends You (2026)
ChatGPT Ranking: Does It Exist? The Honest Answer (2026)
