AI bot types
The three kinds of AI bots that visit websites, training crawlers, search indexers and live fetchers, and which ones affect AI citations and visibility.
Not every AI bot does the same job. Knowing which is which tells you what to allow.
| Type | What it does | Examples | Affects whether AI cites you? |
|---|---|---|---|
| Search index | Builds the index an AI engine searches when answering. | OAI-SearchBot, PerplexityBot, Claude-SearchBot, Googlebot, Bingbot | Yes. Block it and that engine can’t find your pages. |
| Live fetch | Opens a page in real time because someone asked about it. | ChatGPT-User, Perplexity-User, Claude-User | Yes. Block it and the engine can’t read your page mid-answer. |
| Training | Collects content to train future models. | GPTBot, ClaudeBot, Google-Extended, CCBot | No. Blocking it doesn’t stop engines citing you. |
What to allow
Section titled “What to allow”- Allow search index and live fetch bots for every engine you care about.
- Training bots are your choice. Block them if you don’t want your content used for training; it won’t affect citations.
Google and Bing are special
Section titled “Google and Bing are special”Googlebot feeds Google AI Overviews and AI Mode, and Bingbot feeds Copilot and ChatGPT search. Blocking either affects those AI surfaces as well as normal search.
Check your site
Section titled “Check your site”Technical → Bot access shows robots.txt and a live request result for each bot. If one is blocked, see AI crawlers show nothing, or a bot is blocked.