November 13, 2026 · A joint test by Miraqo and GioContextAi

We served llms.txt to AI crawlers for six weeks. None of them asked for it.

One e-commerce site, split in two. Half of its categories went into llms.txt and the other context files for AI engines, half stayed out, and for six weeks we watched both the server log and the citations. The answer turned out to be simpler than the question.

1. The question and how we set it up

GioContextAi is a WordPress plugin that generates context files for AI engines: llms.txt, llms-full.txt, ai.txt and, for WooCommerce stores, products.json. The promise behind files like these is that an AI engine reads them, understands the site better and cites it more often. Giovanni, who builds the plugin, wanted to know whether that holds. So did we: Miraqo measures how often sites get cited in AI answers, and “should I add llms.txt?” is one of the questions we hear most.

The question we wrote down before starting was: does making an AI context file available change whether, and how often, a site is cited by ChatGPT, Gemini, Google AI Overviews and AI Mode?

One site, split in two

Comparing two different sites tells you mostly how those sites differ. So we used one: an Italian e-commerce store for collectors, with a catalogue of tens of thousands of products. We took twelve product categories that had received the same SEO work in the same period and paired them by type: two pairs of stamps by issuing country, one pair of catalogues, two pairs of storage supplies, one pair of coins. That makes six pairs, and from each pair one category went into the context files and the other stayed out.

The two halves share the domain, the history, the server and the CDN rules. Anything that moves the whole site, such as a change in its overall authority, a server problem or a site-wide algorithm update, moves both halves, and the comparison cancels it out.

What a shared domain does not equalise is everything that belongs to the single category: the keywords it ranks for and where, its backlinks, internal links, search demand and the content on the page. The baseline shows it: the “out” half was cited more often than the “in” half from the first round. So we never compared the two halves directly. We compared how the gap between them changed after activation, against how much that gap moved from week to week before it.

The plugin can include or exclude content one taxonomy term at a time, which is what made the split possible. Parent categories and the home page, which list products from both halves, were kept out of the files, so the context never pointed to a page that leads to both sides.

What we measured

• Citations. Sixteen prompts, eight per half, each written around one category. Every week, Miraqo sent each prompt three times to each of the four engines and recorded whether the answer cited a URL from that category. Two prompts (a pair) were removed before activation: in the baseline they were already citing pages from both halves, which would have made them useless as a comparison. That left 7 prompts per half, so 28 prompt–engine cells per half each week, 84 runs.
• Access to the files. The server access log, day by day: every request to the four files, by user agent, and every request from AI crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Applebot, PerplexityBot, Perplexity-User) to any URL. A user agent is only a claim, so we kept a request as coming from a crawler only when its IP address was on the list that crawler’s operator publishes. The log was our primary source because it does not depend on the tool being tested. The plugin’s own dashboard was the cross-check.
• Background noise. The number of products in each category, read every week from the public category pages, because the store owner edits the catalogue daily.

Timeline: three weekly baseline rounds (11, 18 and 25 September), plugin activated with the split on 25 September, then six weekly rounds up to 6 November. The prompts had been tracked since early August, so the baseline also had a month of less regular history behind it.

Three outcomes, written down in advance

1. The half in the context files improves more than the half outside, beyond the week-to-week variation of the baseline.
2. No difference.
3. No AI engine accesses the context files at all.

We agreed to publish whichever came out. It was the third.

2. What happened

Nobody asked for the files

In six weeks, from 25 September to the morning of 6 November, no AI crawler requested llms.txt, llms-full.txt, ai.txt or products.json. The log has no gaps.

The requests the files did get came from our own weekly checks, scanners rotating IP addresses behind browser user agents, a few SEO and web-survey bots, and a script that kept querying products.json as if the site were a Shopify store. The GioContextAi dashboard showed the same thing every week: no requests from AI crawlers. The two sources agreed for all six weeks.

AI crawlers were not absent, though. Over the same period they visited the site tens of thousands of times:

They fetched category pages and product pages. Not once did they fetch the files written for them.

Giovanni explains in section 3 why checking the addresses matters: on another site, close to half the requests calling themselves AI crawlers came from scanners borrowing the name. Here impostors were rare. About 360 requests out of more than 180,000, or 0.2%, declared an AI user agent from an address outside the lists. Among them were the only three Perplexity-User requests of the period, so Perplexity’s agent never actually showed up. Since the lists are recent snapshots, a genuine crawler on an address that has since been dropped would also land in that 0.2%.

The only AI agent that ever asked for one of those paths did so before the test began. On 12 and 13 September, two weeks before the plugin was switched on, Claude-User requested products.json?limit=… four times, from addresses on Anthropic’s list, the way you would query a Shopify store, and got a 404. Once the file existed, nobody came back for it.

Did the crawlers favour the categories in the files?

With the split there was one more thing to check: whether ChatGPT-User, the agent that fetches pages while ChatGPT is writing an answer, visited the categories listed in the files more than the others.

The share going to the “in” half fell, from 78% to 66%. Some of these visits were triggered by our own weekly ChatGPT checks, which fetch the pages they cite, so this is a small and noisy signal. Nothing in it points to the files steering visits.

Citations, week by week

Cells cited, out of 28 per half (a cell counts if at least one of its three runs cited the category):

On cells, the gap moved in favour of the “in” half: from −3.0 on average in the baseline to −0.7 during the test, so +2.3 cells out of 28. Taken alone, that could look like a small effect. It does not hold up:

• The “in” half was already rising during the baseline (6 → 8 → 11), before the plugin was switched on. Its average during the test, 9.8, is below its last pre-activation round.
• On individual runs, the weekly gap during the test ranged from −9.5 to +8.1 percentage points. The whole baseline range (−8.6 to −4.1) sits inside it. The difference between the two halves never left the normal week-to-week noise.
• Above all, there is no channel. If no engine reads the file, the file cannot change what the engine writes.

The background stayed still

The catalogue barely moved: over the whole period the “in” half changed by +0.2% in products, the “out” half by +0.6%. The context files themselves were stable too: llms.txt and ai.txt did not change apart from one service page added on 30 October, and llms-full.txt changed only for the weekly text of a few store pages. We checked the split every week. It held.

3. From preparing content to observing AI crawlers

By Giovanni, developer of GioContextAi

When I started building GioContextAi, the goal was to give WordPress sites a simple way to organise information meant for AI systems and make it available.

Files like llms.txt, llms-full.txt and ai.txt, together with products.json for WooCommerce, answer that need. But during development another question turned out to be just as important: once these resources are published, how can we tell what AI agents actually do on our site?

That question led to AI Crawler Intelligence, a part of GioContextAi dedicated to observing requests from user agents associated with AI systems.

The aim is not just to count visits, but to help whoever runs a site understand which agents say they are accessing it, what kinds of resources they request and how their activity changes over time.

Even in the free version, AI Crawler Intelligence groups requests by type of activity (training, AI search, user-initiated access and others), following the documentation published by the providers. It also shows the daily trend, the types of resources requested and the HTTP response classes, separating successful requests, redirects and errors.

Requests to the context files generated by GioContextAi can be viewed on their own, so you can compare the files being technically available with the activity actually observed.

The Pro version extends the analysis to individual WordPress content: posts, pages, WooCommerce products and taxonomies. An e-commerce manager, for example, can see which products receive the most requests, which crawlers are involved and how that changes over time.

This information does not tell you which products are recommended in AI answers, but it lets you ask more precise questions. Are some sections of the catalogue visited more than others? Do particular crawlers show interest in specific content? Does the frequency of requests change after a technical change?

One aspect is fundamental, though: how reliable the data is. AI Crawler Intelligence recognises crawlers by the user agent they declare in their HTTP requests, and that is not the same as verifying who they are.

A separate analysis of the server logs of GioSuite, a single site with modest traffic, carried out in October 2026, made the problem clear. When the IP addresses were checked with the providers’ official methods, just over half of the requests declaring an AI user agent actually came from verified crawlers. The rest came from two scanners using various crawler names to look for configuration files and credentials. The plugin was fooled too: about a third of the requests recorded in its statistics during that period came from these scanners.

In the same period, llms.txt was requested by verified crawlers during broader crawls of the site, which links to the file from several pages. There was no sign of it being sought out on its own.

These data show both what observation is good for and where it stops. The plugin does not record IP addresses or full URLs, and it only sees requests that reach WordPress, not those handled entirely by a cache, CDN or firewall. Its statistics are therefore a part of the traffic, not a complete record of access to the server.

That points to concrete directions for development: verifying crawler identity, improving error diagnostics and, where possible, bringing in information from infrastructure logs.

The value of AI Crawler Intelligence today is to make part of AI agents’ activity observable and interpretable. The next step is to use that information to spot problems, assess changes and make better-informed technical decisions.

So GioContextAi’s path is widening: not only preparing content for AI systems, but building the tools to observe how those systems interact with your site.

4. What this does and does not tell you

What we can say: on this site, over these six weeks, AI crawlers were active and none of them requested the context files. Citations of the categories in the files did not change in any way that could be told apart from normal variation, and with no file reads there was no route for an effect to take.

Giovanni’s logs from GioSuite, in section 3, point the same way from the other side. There, verified crawlers did fetch llms.txt, but only while crawling the site, because several pages link to it, and nothing suggested they went looking for it. On this site no page and no robots.txt linked to the files, and nobody asked for them.

What we cannot say:

• That llms.txt never works. This is one site, one sector, one language, six weeks of test after a three-week baseline. A different site, or the same one in six months, could see different crawler behaviour.
• What happens if an engine starts reading these files. We measured “a file served to whoever asks for it”. If an engine starts asking, the question has to be measured again.
• That the two halves were equivalent. They were matched by type and by the SEO work they had received, but each category has its own rankings, links and demand, and the baseline shows they did not start level. The comparison handles that by looking at the change in the gap, not at the gap itself, but a three-week baseline is a short one.
• That excluding a category does not hurt it. Our control was “category left out of the file”, not “site without the plugin”. If an engine used llms.txt as a map, being left out could have penalised the control half. Here that limit weighs little: nobody read the map.
• Anything general about content quality. The prompts were written around categories that had already been optimised, so the test describes what a context file adds on ground that was already prepared.

The practical answer to “should I add llms.txt?” is that, on this evidence, it costs little and does not move AI citations, because the engines we measured do not fetch it. If you want to know whether it matters on your own site, the cheapest check is the one we started from: look in your server log for requests to /llms.txt. On this site, after six weeks, there were none.