# AI crawlers show nothing, or a bot is blocked

> A runbook for when GetIntel's technical scan shows an AI bot blocked or AI crawlers shows no visits, with the checks and curl commands to find the cause.

Source: https://getintel.ai/docs/academy/troubleshooting/bot-blocked/

If **Technical → Bot access** shows a bot as blocked, or [AI crawlers](/features/ai-traffic/) shows no visits from an engine, work through this runbook.

## How GetIntel tests bots

The technical scan requests your site as each bot, using the bot's real user agent, and also reads your robots.txt. A request counts as **blocked** when it gets status 401, 402, 403, 406, 429, 451 or 503, or a challenge page such as "Just a moment…" or a captcha.

| Bot | Engine | Role |
| --- | --- | --- |
| OAI-SearchBot | ChatGPT | Search index |
| ChatGPT-User | ChatGPT | Live fetch |
| PerplexityBot | Perplexity | Search index |
| Perplexity-User | Perplexity | Live fetch |
| Claude-SearchBot | Claude | Search index |
| Claude-User | Claude | Live fetch |
| Googlebot | Google (AI Overviews and AI Mode) | Search index |
| Bingbot | Copilot and ChatGPT search | Search index |

**Training bots** (GPTBot, ClaudeBot, Google-Extended, CCBot) are listed separately. Blocking them does not stop AI engines from citing you.

## 1. Check robots.txt

Open `https://yoursite.com/robots.txt`. Look for a `Disallow: /` under the bot's name, or under `User-agent: *` without a later allow for the bot.

## 2. Request your site as the bot

```sh
curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot" \
  https://yoursite.com/
```

- `200`: the bot gets through.
- `403`, `429` or `503`: something in front of your site is blocking it.

To see a challenge page, fetch the body and look for "Just a moment" or "captcha":

```sh
curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \
  https://yoursite.com/ | grep -i -E "just a moment|captcha|challenge"
```

## 3. Check your firewall or CDN

The most common cause is a bot-protection setting:

- **Cloudflare:** check **Security → Bots** (Bot Fight Mode, and any setting that blocks AI bots) and **Security → WAF** custom rules. Allow the search and live-fetch bots above.
- **Other CDNs and hosts:** look for bot management, rate limiting or "block AI crawlers" options.

<Aside type="caution">
Many firewalls block anything claiming to be Googlebot or Bingbot unless the request comes from Google's or Microsoft's own IP addresses. GetIntel can't verify those two from outside, so a "blocked" result for them may not reflect what the real bot sees.
</Aside>

## 4. AI crawlers shows nothing

- **Not connected:** AI crawlers needs [Cloudflare](/docs/integrations/cloudflare/). Check **Integrations**.
- **Just connected:** the first sync takes a few minutes, then updates hourly.
- **Wrong website:** the Cloudflare token must be for the zone that serves your site.
- **Bots are blocked:** blocked bots never reach your pages, so there is nothing to count. Fix steps 1 to 3 first.

## 5. Re-check

After a fix, open **Technical** and choose **Re-check now** (once an hour).

## Contacting support

Send us the bot name, the page URL, the output of the curl command above, and whether your site is behind Cloudflare or another CDN.