None of the 37 most-cited pages in the AI visibility tooling category block an AI crawler. Not one. We checked their robots.txt on 18 August 2026 for GPTBot, PerplexityBot, ClaudeBot and three others, and found zero blocks. The gate every access checklist leads with is not the gate anyone is failing.
Disclosure: GetIntel is an AI visibility tool and ships a crawler-access check of its own, so we are auditing a checklist we also sell. The full dataset, including the two checks we threw away, is published alongside this piece.
The standard advice runs: let the bots in, add llms.txt, add schema, make sure the page renders without JavaScript. It is sensible advice and it is cheap to follow. What nobody has done is check it against the pages that actually win citations.
So we did, and the results argue for taking the checklist less literally.
Which pages did we test?
The top-cited page from each of the 39 most-cited domains in the AI visibility tooling category, taken from GetIntel's own tracking of that category: 12,868 cited source URLs across 5,292 AI answers, checked 18 August 2026. 37 returned HTTP 200 and form the sample.
That list is Reddit, YouTube, Semrush, LinkedIn, arXiv, TechRadar, Ahrefs, HubSpot, Zapier, GitHub and the specialist tools that show up alongside them. These are not hypothetical winners. They are the pages AI engines reached for when answering buyer questions in this category.
Pass rates, checked 18 August 2026:
| Gate | Pass |
|---|---|
| Title tag present | 37 of 37 |
| robots.txt present | 36 of 37 |
| Readable without JavaScript | 35 of 37 |
| Meta description present | 34 of 37 |
| sitemap.xml present | 32 of 37 |
| JSON-LD schema present | 32 of 37 |
| llms.txt present | 29 of 37 |
Is anyone actually blocking AI crawlers?
No. Zero of 37 block a named AI bot, and the more useful number is the one underneath it: 27 of the 36 robots.txt files do not mention an AI bot at all.
They neither allow nor deny GPTBot, PerplexityBot or ClaudeBot. They simply do not address them, so the crawlers proceed under default-allow rules written for search engines years ago.
That reframes the advice. "Allow AI crawlers in robots.txt" sounds like an action, but for most sites it describes a state you already have by doing nothing. It is worth checking once, because the failure mode is severe when it happens, and then it is worth forgetting about.
Do the winners have llms.txt?
Often not. 29 of 37 carry one, which means eight heavily-cited pages have no llms.txt at all: Ahrefs, arXiv, SE Ranking, TechRadar, Profound, useomnia, YouTube and ZipTie.
Ahrefs alone was cited 440 times in our tracking, arXiv 596, TechRadar 360. If llms.txt were a prerequisite for citation, those numbers would not exist.
None of this is new. SE Ranking's study across roughly 300,000 domains found no measurable citation effect, Google has said publicly that it does not use the file, and GetIntel's own test across the 50 most-cited domains on 11 August 2026 found the same nothing, which is covered in full in does llms.txt actually do anything. The point here is narrower: llms.txt is one gate of seven, and it is the one the winners fail most often.
What about schema and JavaScript?
32 of 37 pages carry JSON-LD and 35 of 37 render without JavaScript, but the exceptions are the two biggest winners in the category, which makes this the most interesting result in the sweep. The five without it include Reddit, the single most-cited domain in AI visibility tooling at 1,262 citations. Ahrefs, arXiv, SE Ranking and GitHub also ship none on the page we tested.
The JavaScript gate is starker. 35 of 37 pages render enough text to read without JavaScript. The two that do not are Reddit and YouTube, the number one and number two most-cited domains. Stripped of scripts, the Reddit thread yields one word and the YouTube page yields 35.
So the two biggest citation winners in AI visibility tooling fail the gate that says a page must be readable without JavaScript, and one of them also fails the schema gate.
Which specific pages failed what?
Naming them matters, because an aggregate pass rate hides whether the failures land on weak pages or strong ones. Here they land on strong ones. All figures checked 18 August 2026.
| Page (top-cited for its domain) | Citations to domain | Every gate it fails |
|---|---|---|
| reddit.com | 1,262 | No JSON-LD; no meta description; 1 word without JavaScript |
| youtube.com | 1,054 | No llms.txt; no sitemap.xml; 35 words without JavaScript |
| arxiv.org | 596 | No llms.txt; no sitemap.xml; no JSON-LD |
| tryprofound.com | 468 | No llms.txt |
| ahrefs.com | 440 | No llms.txt; no JSON-LD |
| seranking.com | 408 | No llms.txt; no sitemap.xml; no JSON-LD |
| techradar.com | 360 | No llms.txt |
| useomnia.com | 301 | No llms.txt; no meta description |
| ziptie.dev | 227 | No llms.txt |
| github.com | 225 | No sitemap.xml; no JSON-LD |
Seven of those ten pages are cited more often than getintel.ai, which passes all seven gates and sits at 315 citations in the same category. The other three are cited less: useomnia at 301, ziptie at 227, github at 225. That cuts both ways honestly. Passing every gate has not put getintel.ai ahead of Reddit, arXiv or Ahrefs, and failing several has not stopped github or ziptie from landing within a hundred citations of a site that passes them all. Whatever separates them, it is not the checklist.
Why do Reddit and YouTube get cited anyway?
Because engines do not reach them the way they reach your blog.
A crawler fetching your marketing site sees whatever your server returns. Reddit and YouTube content arrives through other routes: licensing arrangements, dedicated APIs, and index coverage built over a decade of being the canonical place a discussion or a video lives. The engine already holds the content before rendering is a question.
That is why their failures do not transfer as permission. If your site returns one readable word without JavaScript, no arrangement exists to compensate for it. The gates describe how an engine reaches an ordinary site, and almost every site is ordinary.
It also explains why the checklist looks so weakly correlated across the AI visibility tooling sample. The sample is dominated by pages that win citations for reasons far upstream of markup: being Reddit, being arXiv, being the publication a journalist already trusts.
Does that mean the gates do not matter?
They matter as a floor, not as a lever, and this is where it would be easy to overclaim in either direction.
Reddit and YouTube are not normal websites. Engines reach their content through direct arrangements, APIs and long-established indexing paths that a SaaS blog does not get. Their presence in an answer says something about the engine's source preferences, not about whether your marketing site can skip rendering.
The honest reading is narrower. These gates are a floor, not a lever. Passing them stops you being invisible for a stupid reason. It does not make you citable, and the evidence that any single one of them raises citation rate is thin to absent.
Where that leaves the checklist: run it once, fix anything genuinely broken, and stop treating a green score as progress. GetIntel's own site scores 96 on the same 12 checks and is named in 5.9 percent of AI visibility category answers. The gates were never the constraint.
What did we get wrong?
Two of the nine checks GetIntel ran, noindex and nosnippet, produced false positives on every hit and were discarded rather than published.
We measured noindex and nosnippet by string matching the fetched HTML. Both produced false positives. One page of 37 appeared to carry noindex and, on re-fetch, had no robots meta tag at all. Four appeared to block snippets, and on inspection one carried max-snippet:-1, which is unlimited snippet, the opposite instruction. Both counts and the flagged domains are recorded in the published dataset so the discard is checkable.
The cause is obvious in hindsight. These are SEO blogs. They write about the directives we were grepping for, so the words appear in body copy. Any audit that string-matches technical directives across SEO content will find them everywhere and be wrong.
We dropped both checks rather than publish numbers we could not reproduce. The dataset records the decision.
How do you run this on your own site?
Four checks, in the order that the failure severity actually justifies.
- Fetch your own robots.txt and read it. You are looking for a
Disallow: /under a block naming an AI bot. This is rare and catastrophic, so check once and move on. - Curl your page with JavaScript off and count the words that survive. If the answer is near zero and you are not Reddit, that is a real problem.
- Check for one JSON-LD block describing your organisation. Entity schema helps an engine identify you, which is a different job from content schema, and the evidence for the two is not the same.
- Add llms.txt if you like, then judge it on nothing. Nobody has shown it moves citations, including us.
Then stop auditing and go read where your citations actually come from, because on this evidence the technical layer is not where the difference is being made. If you want the fixes shipped rather than listed, that path runs through your own coding agent.