AI crawlers show nothing, or a bot is blocked

If Technical → Bot access shows a bot as blocked, or AI crawlers shows no visits from an engine, work through this runbook.

How GetIntel tests bots

The technical scan requests your site as each bot, using the bot’s real user agent, and also reads your robots.txt. A request counts as blocked when it gets status 401, 402, 403, 406, 429, 451 or 503, or a challenge page such as “Just a moment…” or a captcha.

Bot	Engine	Role
OAI-SearchBot	ChatGPT	Search index
ChatGPT-User	ChatGPT	Live fetch
PerplexityBot	Perplexity	Search index
Perplexity-User	Perplexity	Live fetch
Claude-SearchBot	Claude	Search index
Claude-User	Claude	Live fetch
Googlebot	Google (AI Overviews and AI Mode)	Search index
Bingbot	Copilot and ChatGPT search	Search index

Training bots (GPTBot, ClaudeBot, Google-Extended, CCBot) are listed separately. Blocking them does not stop AI engines from citing you.

1. Check robots.txt

Open https://yoursite.com/robots.txt. Look for a Disallow: / under the bot’s name, or under User-agent: * without a later allow for the bot.

2. Request your site as the bot

curl -s -o /dev/null -w "%{http_code}\n" \
  -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot" \
  https://yoursite.com/

• 200: the bot gets through.
• 403, 429 or 503: something in front of your site is blocking it.

To see a challenge page, fetch the body and look for “Just a moment” or “captcha”:

curl -s -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \
  https://yoursite.com/ | grep -i -E "just a moment|captcha|challenge"

3. Check your firewall or CDN

The most common cause is a bot-protection setting:

• Cloudflare: check Security → Bots (Bot Fight Mode, and any setting that blocks AI bots) and Security → WAF custom rules. Allow the search and live-fetch bots above.
• Other CDNs and hosts: look for bot management, rate limiting or “block AI crawlers” options.

Caution

Many firewalls block anything claiming to be Googlebot or Bingbot unless the request comes from Google’s or Microsoft’s own IP addresses. GetIntel can’t verify those two from outside, so a “blocked” result for them may not reflect what the real bot sees.

4. AI crawlers shows nothing

• Not connected: AI crawlers needs Cloudflare. Check Integrations.
• Just connected: the first sync takes a few minutes, then updates hourly.
• Wrong website: the Cloudflare token must be for the zone that serves your site.
• Bots are blocked: blocked bots never reach your pages, so there is nothing to count. Fix steps 1 to 3 first.

5. Re-check

After a fix, open Technical and choose Re-check now (once an hour).

Contacting support

Send us the bot name, the page URL, the output of the curl command above, and whether your site is behind Cloudflare or another CDN.