ClaudeBot is Anthropic's web crawler. It collects public pages that may be used to train future Claude models. On getintel.ai it made 607 IP-verified requests in the 30 days to 15 October 2026, and 92% of them came on just four days. Claude-User, the agent that fetches a page when someone asks Claude a question, came every single day.
Disclosure: GetIntel sells AI visibility tracking, including AI crawler logs. These numbers come from one site, ours, over one month, from DataFast's bot report. The daily counts are downloadable as a CSV. For every AI bot side by side, see our full AI crawler log.
In this article
- What is ClaudeBot?
- How often does ClaudeBot crawl?
- What does ClaudeBot read?
- Claude-User is the one that answers people
- Claude asks for the URL without the trailing slash
- Where was Claude-SearchBot?
- How to allow or block ClaudeBot
- Should a B2B company block ClaudeBot?
- What ClaudeBot's logs can't show
What is ClaudeBot?
ClaudeBot is Anthropic's training crawler. Claude-User fetches pages live for Claude users, and Claude-SearchBot crawls to improve Claude's search results. Each has its own robots.txt token:
| Agent | What it does | Requests to getintel.ai, 30 days |
|---|---|---|
ClaudeBot | Collects public pages for training future models | 607 |
Claude-User | Fetches a page when a Claude user's question needs it | 172 |
Claude-SearchBot | Crawls to improve Claude's search results | 0 |
Anthropic's agents made 779 of the 17,599 verified bot requests our site got in that window (4.4%). That put Anthropic seventh of the nine companies in our logs, well behind OpenAI's 5,408. ClaudeBot on its own made 3.4% of all bot requests.
ClaudeBot's burst pattern means a heavy day in your logs is normal, not a sign of steady load. Anthropic's crawler documentation says its bots respect robots.txt, including the non-standard Crawl-delay line.
How often does ClaudeBot crawl?
In bursts. ClaudeBot made 560 of its 607 requests on four days between 16 September and 15 October 2026, and none at all on 16 days.
Daily requests by agent. Chart by GetIntel from DataFast bot logs for getintel.ai, October 2026.
ClaudeBot's month came in four bursts:
| Day | ClaudeBot requests |
|---|---|
| 20 September | 312 |
| 22 September | 132 |
| 12 October | 78 |
| 4 October | 38 |
| Other 26 days combined | 47 |
The four busiest days carried 560 of its 607 requests. Every other day it sent 0 to 19. It's not a crawler that trickles through your site. It arrives, reads a large slice in one go, and leaves for a week or more.
What does ClaudeBot read?
ClaudeBot made 605 requests we could match to a page, and almost all were first visits to nearly 600 distinct pages. Only the homepage (3 requests) and one guide (2) were fetched more than once at the same URL. That is about 1.0 requests per page; the next lowest was GPTBot at 1.1.
By section:
| Section | Share of ClaudeBot requests |
|---|---|
| Product docs | 43% |
| Blog | 26% |
| AI Software Index | 21% |
| Solutions pages | 3% |
| Features pages | 2% |
| Homepage | 1% |
That's a training crawler's diet: long, structured reference pages with stable URLs. OpenAI's, Perplexity's and Bing's search crawlers spent 35% to 42% on the blog instead. If you want a future Claude model to describe your product accurately, your docs are the pages ClaudeBot reads most. Make sure they say plainly what the product does.
A single pass also means timing matters. A page that was wrong on the day of a burst stays wrong in that crawl until the next one.
Claude-User is the one that answers people
Claude-User made 172 requests, at least 2 on every day of the 30, and never more than 14. It's steady because it follows users, not a crawl schedule.
What it fetched also looks like a buyer's questions, not a training sweep:
| Page | Claude-User fetches |
|---|---|
| Best SaaS directories | 19 |
| AirOps pricing | 12 |
| What AI visibility tools cost an agency | 11 |
| Homepage | 11 |
| How much AI visibility monitoring should cost | 5 |
| AI visibility for agencies | 5 |
Counts are page loads after redirects (see the next section). Pricing and agency pages dominate. In the whole month Claude-User loaded 23 distinct pages, against ClaudeBot's nearly 600. So when Claude names you in an answer, it's reading a handful of pages, not your whole site.
Claude asks for the URL without the trailing slash
80 of Claude-User's 172 requests were for a URL without its trailing slash. Our site answers those with a 308 redirect to the slash version, and Claude followed it every time, so most of the 172 are pairs: the slashless request, then the page load. Apart from the homepage, every page Claude-User loaded arrived this way.
We can't see why from the logs. One possibility is that Claude builds the URL itself in a normalised form rather than copying the exact link. ClaudeBot didn't show the pattern; it requested canonical URLs.
The practical check takes a minute. Request your most important pages without the trailing slash, or with one if your site doesn't use them, and confirm you get a single redirect to a 200. A 404 on the variant, a redirect chain, or a firewall rule that challenges the second request would fail almost half of Claude's live fetches. Your robots.txt could be perfect and Claude would still be unable to read the page.
Where was Claude-SearchBot?
Claude-SearchBot made no verified requests to getintel.ai in the 30 days to 15 October 2026. Anthropic documents Claude-SearchBot as the crawler that improves search results inside Claude, but none came. Claude's answers that touched our pages came through Claude-User fetching them live.
That could change, and it's a reason not to write rules for "the Anthropic crawler" as if there were one. Name all three tokens.
How to allow or block ClaudeBot
Allowing it needs nothing. If your robots.txt doesn't mention ClaudeBot and your firewall doesn't block it, it can crawl. Most sites in our access test never mention AI bots at all.
To block training but keep Claude able to read your pages for users:
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
To slow ClaudeBot down instead of blocking it:
User-agent: ClaudeBot
Crawl-delay: 5
To block all three, use a Disallow: / group for each token. Blocking ClaudeBot alone doesn't stop Claude-User. Each agent reads only its own group (or the * group if it has none).
Two things robots.txt can't fix. A CDN or bot-protection product can block Anthropic's agents before robots.txt matters, so check your firewall's bot settings too. And robots.txt only works going forward. Pages ClaudeBot has already read stay read.
Should a B2B company block ClaudeBot?
No, not for most B2B software companies. Blocking ClaudeBot removes your docs and pages from what future Claude models learn about your category, which is the opposite of what you want if buyers ask Claude which tool to use. It changes what future models learn, not what Claude-User fetches today, and we saw Claude-User every day.
If ClaudeBot's bursts strain your server, Crawl-delay is gentler than a block. If you sell your content, blocking ClaudeBot while allowing Claude-User keeps you readable in answers without contributing to training.
What ClaudeBot's logs can't show
- It's one site over one month. 607 ClaudeBot requests is a small sample. The burst pattern is clear, but how often bursts happen will vary by site.
- A fetch isn't a mention. Claude-User loading a page means Claude read it for an answer, not that the answer named us.
- Purpose labels are DataFast's. We used its classification and its IP verification. Requests that only claimed to be ClaudeBot from other IP ranges are excluded.
- We can't see the questions. The fetched pages hint at buyer intent, but the logs don't contain what users asked.
GetIntel's AI traffic view shows the same per-agent split for your own site, from your Cloudflare and GA4 data.