OpenAI's search crawler visited getintel.ai 3,326 times in the 30 days to 15 October 2026. Googlebot came 1,957 times. That's the headline from a month of IP-verified bot logs on our own site: OAI-SearchBot out-crawled Googlebot by 1.7 to 1. And OpenAI's three agents together made 31% of every verified bot request we received.
In the same 30 days, only 10% of bot traffic (1,792 of 17,599 requests) was an AI assistant fetching a page to answer someone right then. Most training bots read each page once or twice and leave. Search bots come back three to five times. And ChatGPT, when a user asked it something that led to us, fetched our homepage more than every other page combined.
Disclosure: GetIntel sells AI visibility tracking, including AI crawler logs. These numbers come from one site, ours, over one month. The per-bot table is downloadable as a CSV so you can check the arithmetic.
In this article
- What did we measure?
- Which AI crawlers visit the most?
- How much of it is AI answering a user right now?
- Most training bots read once, search bots come back
- What does each bot actually read?
- What does ChatGPT fetch when a user asks about you?
- Where are Google's and Perplexity's AI fetchers?
- Which crawl errors showed up in the logs?
- The robots.txt token for each AI crawler
- What this data cannot tell you
What did we measure?
We measured every bot request to getintel.ai from 16 September to 15 October 2026, using DataFast's bot traffic report. We kept only requests whose IP address matched the operator's published ranges, which removes scrapers that just borrow a famous user agent. That left 17,599 requests from 17 user agents run by 9 companies. Volume was steady, at roughly 3,700 to 4,100 requests in each full week.
DataFast also labels every agent by purpose, and we kept its labels:
- Search index: building the index an AI search product or search engine answers from.
- Model training: collecting text to train future models.
- Live answer fetch: an assistant loading a page because a user asked a question right then.
For the page-level numbers below we matched requests to URLs. That match covers 17,262 of the 17,599 requests (98%). The missing 2% is a long tail of single requests.
getintel.ai is a small B2B SaaS site: about 930 distinct URLs got a bot visit, split mainly across a blog, product docs, and the AI Software Index. Your ratios will differ. The patterns below are the part we'd expect to carry over.
Which AI crawlers visit the most?
Requests by user agent, coloured by purpose. Chart by GetIntel from DataFast bot logs for getintel.ai, October 2026.
| Bot | Company | Purpose | Requests | Share |
|---|---|---|---|---|
| OAI-SearchBot | OpenAI | Search index | 3,326 | 18.9% |
| Amazonbot | Amazon | Training | 2,549 | 14.5% |
| PerplexityBot | Perplexity | Search index | 2,141 | 12.2% |
| Bingbot | Microsoft | Search index | 2,045 | 11.6% |
| Googlebot | Search index | 1,957 | 11.1% | |
| ChatGPT-User | OpenAI | Live answer | 1,578 | 9.0% |
| meta-externalagent | Meta | Training | 1,356 | 7.7% |
| GoogleOther | Training | 667 | 3.8% | |
| Applebot | Apple | Training | 623 | 3.5% |
| ClaudeBot | Anthropic | Training | 607 | 3.4% |
| GPTBot | OpenAI | Training | 504 | 2.9% |
| Claude-User | Anthropic | Live answer | 172 | 1.0% |
| DuckAssistBot | DuckDuckGo | Live answer | 33 | 0.2% |
The table shows 13 of the 17 user agents. The other four are Google's (Google-InspectionTool, Google-NotebookLM, GoogleAgent, Google-Read-Aloud), with 41 requests between them.
By company, OpenAI leads with 5,408 requests (30.7%), then Google at 2,665, Amazon at 2,549, Perplexity at 2,141 and Microsoft at 2,045. Anthropic sent 779 and Apple 623.
GPTBot is the OpenAI crawler most robots.txt guides talk about, and it made only 504 of OpenAI's 5,408 requests (9%). The search crawler and the live fetcher did the rest. A site that blocks "OpenAI" by blocking GPTBot has blocked less than a tenth of OpenAI's traffic, and none of the part that puts it in ChatGPT's search answers.
How much of it is AI answering a user right now?
AI answering a user in real time was 10% of bot traffic on getintel.ai: 1,792 of 17,599 requests, 16 September to 15 October 2026. Indexing was 9,501 (54%) and training was 6,306 (36%).
ChatGPT-User alone was 1,578 of those 1,792 live fetches (88%). Claude-User was 172 (about half of them redirect hops, covered below) and DuckAssistBot 33. Google's three live fetchers together made 9.
So "how many times did AI read my site" and "how many times did AI read my site for a user" are numbers about ten times apart. A crawler report that adds them together is mostly showing you index and training traffic. Still, live fetches are the ones closest to a buyer. Each one is a person who asked ChatGPT or Claude a question that led the assistant to your page. That's also why being named and being cited are separate events: a live fetch is the cited half.
Most training bots read once, search bots come back
Most training bots read each URL once or twice in the month; search bots re-read each one three and a half to nearly five times. Amazonbot is the exception. Divide each bot's requests by the distinct URLs it touched and the two groups separate:
| Bot | Purpose | Distinct URLs | Requests per URL |
|---|---|---|---|
| ClaudeBot | Training | 597 | 1.0 |
| GPTBot | Training | 433 | 1.1 |
| Applebot | Training | 371 | 1.7 |
| GoogleOther | Training | 361 | 1.8 |
| meta-externalagent | Training | 607 | 2.2 |
| Amazonbot | Training | 722 | 3.5 |
| Bingbot | Search index | 573 | 3.5 |
| PerplexityBot | Search index | 521 | 4.1 |
| OAI-SearchBot | Search index | 697 | 4.4 |
| Googlebot | Search index | 412 | 4.7 |
ClaudeBot read 597 URLs in 605 matched requests. It reached nearly as many pages as OAI-SearchBot but visited each one about once. Most of that came in four bursts, which our ClaudeBot breakdown covers day by day. The search indexers came back three and a half to nearly five times per page in the same month, which is what you'd expect from a crawler that has to keep answers fresh.
In practice: a page you update today is likely to be re-read by ChatGPT's and Perplexity's search bots within about a week, since they revisited pages three and a half to nearly five times a month. A training crawler that read the old version this month may not come back for a long time. Amazonbot sits between the groups. It's labelled training, but it re-crawls like an indexer and covered more URLs (722) than any other bot. Our Amazonbot breakdown shows it also crawls about twice as hard at weekends.
What does each bot actually read?
Most training bots spent the bulk of their requests on our product docs, while most search bots spent theirs on the blog. Split by site section:
| Bot | Blog | Docs | AI Software Index | Homepage |
|---|---|---|---|---|
| OAI-SearchBot | 35% | 15% | 25% | 12% |
| PerplexityBot | 36% | 19% | 6% | 3% |
| Bingbot | 42% | 23% | 10% | 3% |
| Googlebot | 18% | 9% | 6% | 25% |
| GPTBot | 11% | 49% | 22% | 1% |
| ClaudeBot | 26% | 43% | 21% | 1% |
| Applebot | 20% | 52% | 1% | 10% |
| GoogleOther | 14% | 69% | 2% | 2% |
| meta-externalagent | 15% | 45% | 8% | 7% |
All five training crawlers in the table spent 43% to 69% of their requests on product docs. Amazonbot, not in the table, is the exception: 30% blog, 26% docs, 21% AI Software Index. Bingbot, PerplexityBot and OAI-SearchBot spent the most on the blog, while Googlebot went to the homepage first (25%). Docs are long, structured, factual pages with stable URLs, which is what you'd keep for training data. The blog is where the comparison and "best tools" pages sit, and those are what a search answer quotes.
If you want future models to describe your product accurately, your docs are the pages training bots read most. If you want to be named in ChatGPT's or Perplexity's answers this month, the search indexers are reading your blog. Applebot is the odd one in the table, with a tenth of its requests on the homepage; our Applebot breakdown covers why, and what Applebot-Extended controls. meta-externalagent read the same docs-first mix in a single nine-day sweep; our meta-externalagent breakdown has the days.
What does ChatGPT fetch when a user asks about you?
ChatGPT-User fetched 199 distinct URLs, but 53% of its 1,578 requests were for the homepage (838). Next came the blog at 33% and docs at 6%.
Its most-fetched pages after the homepage:
| Page | ChatGPT-User fetches |
|---|---|
| Peec AI pricing | 70 |
| Best SaaS directories | 63 |
| Semrush AI Visibility Toolkit pricing | 37 |
| Profound pricing | 32 |
| AI search statistics | 30 |
| Best GEO tools | 29 |
| Otterly pricing | 24 |
Our pages on what other vendors charge took 266 fetches in all (the rows above plus smaller ones), 17% of everything ChatGPT-User loaded. People ask ChatGPT what a tool costs, and ChatGPT goes and reads a page that answers that plainly.
Our reading of the homepage share: when a user asks about a company, the assistant goes to the homepage first. If the homepage doesn't say in plain text what you do, who it's for and what it costs, that fetch gets the assistant very little. Live fetches only happen inside the real assistant, which is one reason the ChatGPT API and the real chatgpt.com interface cite such different sources. How OpenAI's three bots differ from each other, day by day, is in our GPTBot vs OAI-SearchBot vs ChatGPT-User breakdown.
Where are Google's and Perplexity's AI fetchers?
Neither showed up much. In the 30 days to 15 October 2026, Google's live AI agents made 9 requests to getintel.ai, and Perplexity's live fetcher made none we could verify. PerplexityBot, its indexer, made 2,141.
Google. Google-NotebookLM, GoogleAgent and Google-Read-Aloud made 9 requests between them. There's no separate Google crawler for AI Overviews or Gemini answers. Those draw on the normal Googlebot index. "Google-Extended" is a robots.txt token that controls whether Googlebot-crawled content can be used for Gemini training. It never appears in a log, because no bot sends it as a user agent.
Perplexity. PerplexityBot made 2,141 requests, all of them indexing. We logged no verified Perplexity-User requests at all. Either Perplexity answered from its own index, or its live fetches didn't come from IP ranges DataFast could verify. Either way, the indexer is the Perplexity bot you can see and the one to keep unblocked. Our PerplexityBot breakdown shows which pages it re-read every few days.
Which crawl errors showed up in the logs?
Two, and both are cheap to fix: bots kept requesting URLs that no longer existed, and Claude asked for pages without their trailing slash.
Bots keep asking for dead URLs for weeks. Our docs used to link to Markdown copies of 44 API pages that didn't exist. We fixed that on 26 September. Those URLs were still collecting 404s after the fix, 263 of them over the month, from nine different bots. OAI-SearchBot got the most errors of any bot, 178 (5.4% of its 3,326 requests). Blog posts we deleted on 19 September kept getting requests too. If you delete or move pages, redirect them. A crawler's queue of URLs to recheck outlives your sitemap.
Claude asks for URLs without the trailing slash. 80 of the 171 Claude-User requests we matched to a page (of 172 in all, 47%) were for a URL without the trailing slash. Each one got a 308 redirect, then a second request for the real page. The redirect works, so nothing broke. But a site where the slashless URL returns 404, or a redirect a firewall blocks, would fail almost half of Claude's live fetches. Check that both forms of your important URLs resolve.
The robots.txt token for each AI crawler
Every bot below has its own robots.txt token. Blocking one company's crawler doesn't block its other agents, so name each one separately. Live fetchers act on a user's request, and operators handle robots.txt for them differently from their crawlers.
| Token | What it controls | Block it and you lose |
|---|---|---|
OAI-SearchBot | ChatGPT search index | Appearing in ChatGPT search answers |
ChatGPT-User | ChatGPT live fetches | ChatGPT reading your page for a user |
GPTBot | OpenAI model training | Future OpenAI models learning about you |
ClaudeBot | Anthropic model training | Future Claude models learning about you |
Claude-User | Claude live fetches | Claude reading your page for a user |
PerplexityBot | Perplexity index | Appearing in Perplexity answers |
Amazonbot | Amazon crawling, possibly AI training | Amazon using your pages in its products (Amzn-SearchBot and Amzn-User are separate tokens) |
meta-externalagent | Meta AI training | Meta AI models learning about you |
Applebot-Extended | Apple model training (token only) | Apple Intelligence training; Applebot still crawls for Siri and Spotlight |
Google-Extended | Gemini training (token only) | Gemini training use; AI Overviews still run on Googlebot |
For a B2B company that wants to be recommended, the search and live-fetch tokens are the ones you never want blocked. Training is a real choice, and blocking it costs you nothing in this month's answers. Sites with paid or licensed content often block the training tokens; a software company that wants future models to know what it does usually leaves them open. A firewall or bot-protection rule that blocks by default can do the damage anyway, even with a perfect robots.txt. That's the first thing our access checklist test checks for.
What this data cannot tell you
- It's one site. A docs-heavy SaaS site attracts training bots to its docs. A media site would look different. Treat the ratios as one data point.
- It's one month. Crawl rates change when an operator changes its crawler, and new agents appear. Claude-SearchBot, for example, didn't show up in our logs at all.
- Purpose labels are DataFast's. We didn't reclassify anything. Amazonbot in particular behaves partly like an indexer.
- A fetch isn't a citation. ChatGPT-User loading a page means the page was read for an answer, not that the answer named or linked us.
- Unverified traffic is excluded. Bots that spoof a user agent from other IP ranges are left out on purpose, so total bot traffic was higher than 17,599.
To see the same split on your own site, GetIntel's AI traffic view (Pro and Growth plans) shows which AI crawlers fetched which pages and which visitors arrived from AI answers, using your own Cloudflare and GA4 data.