meta-externalagent is Meta's AI crawler, which Meta says trains its foundation AI models and indexes content for its products. In our DataFast logs of IP-verified bot requests to getintel.ai, 16 September to 15 October 2026, it made 1,356 requests, and 93% came in one nine-day sweep, 28 September to 6 October.
None of Meta's four other agents appeared as verified requests in the same logs: meta-webindexer (citations in Meta AI's answers), meta-externalfetcher (loads pages for users), facebookexternalhit (link previews) or meta-externalads.
Disclosure: GetIntel sells AI visibility tracking, including AI crawler logs. These numbers come from one site, ours, over one month, from DataFast's bot report. The daily counts are downloadable as a CSV. For every AI company's bots side by side, see our full AI crawler log.
In this article
- What are Meta's crawlers?
- One nine-day sweep, then almost nothing
- Meta's was the longest sweep
- What meta-externalagent read
- Why meta-externalagent requested tracked homepage URLs
- The robots.txt rules for Meta's bots
- Should you block meta-externalagent?
- What our Meta logs could not see
What are Meta's crawlers?
Meta documents five crawlers, and only one of them says it trains models. On getintel.ai, 16 September to 15 October 2026, that was the only one we logged:
| User agent | What Meta says it does | Requests to getintel.ai, 30 days |
|---|---|---|
meta-externalagent | Trains foundation AI models, or improves products by indexing content | 1,356 |
meta-webindexer | Improves Meta AI search results, and helps Meta AI cite and link to your pages | 0 |
meta-externalfetcher | Fetches individual links a user asked for, including agentic tasks | 0 |
facebookexternalhit | Builds the title, description and thumbnail when a link is shared | 0 |
meta-externalads | Crawls for advertising and business products | 0 |
Meta made 1,356 of the 17,599 verified bot requests our site received (7.7%), which put it sixth of the nine companies in our logs, between Microsoft (2,045) and Anthropic (779).
One nine-day sweep, then almost nothing
meta-externalagent made 1,256 of its 1,356 requests between 28 September and 6 October 2026. Its busiest day was 30 September, with 401.
Daily meta-externalagent requests, days in IST, the sweep highlighted. Chart by GetIntel from DataFast bot logs for getintel.ai, October 2026.
| Period | Requests | Per day |
|---|---|---|
| 16 to 27 September (12 days) | 92 | 8 |
| 28 September to 6 October (9 days) | 1,256 | 140 |
| 7 to 15 October (9 days) | 8 | under 1 |
It made no requests at all on 8 of the 30 days, and after the sweep it went almost silent: 6 requests on 7 October, then one each on 12 and 15 October. Inside the sweep it never dropped below 31 a day.
Meta's was the longest sweep
On getintel.ai, GPTBot, ClaudeBot, Applebot and meta-externalagent each made a concentrated pass between 19 September and 12 October 2026, and Meta's was the longest at nine days. Amazonbot, another training crawler, crawled steadily instead:
| Crawler | Main sweep | Share of its month |
|---|---|---|
| GPTBot | 19 and 24 September | 90% |
| ClaudeBot | 20 and 22 September, 4 and 12 October (four days) | 92% |
| Applebot | 23 to 27 September | 61% |
| meta-externalagent | 28 September to 6 October | 93% |
The sweeps for GPTBot, ClaudeBot and Applebot each have their own breakdown. We can't tell from one site whether the timing means anything, or whether it is a reaction to the 20 old posts we deleted on 19 September. Pages you update after a sweep ends may not be re-read until Meta's next pass, which Meta does not schedule publicly.
What meta-externalagent read
On getintel.ai, meta-externalagent spent 45% of its requests (16 September to 15 October 2026) on documentation pages, more than on any other section.
| Section | Share of meta-externalagent requests |
|---|---|
| Docs | 45% |
| Features, comparisons and other pages | 18% |
| Blog | 15% |
| AI Software Index | 8% |
| Glossary | 3% |
| Tools | 3% |
| Homepage (including tracked-link versions) | 7% |
Docs took 45% of its requests, and 283 of them (21%) were the raw .md copies of our docs pages that we publish for machines. Docs were also the largest section for GPTBot (49% of its requests), ClaudeBot (43%) and Applebot (52%).
It also re-read more than GPTBot, ClaudeBot or Applebot (about one to 1.7 requests per URL each). It touched 607 distinct URLs at 2.2 requests each. 286 of its pages were read once and 150 twice.
Why meta-externalagent requested tracked homepage URLs
Because it followed links exactly as directory sites print them. On getintel.ai, 62 of meta-externalagent's 100 homepage requests (16 September to 15 October 2026) carried tracking parameters from directory listings, such as ?ref=taaft and ?utm_source=comingup.io. Googlebot requested the same tagged URLs. That is harmless as long as your pages carry a canonical tag, so the tagged copies fold back into the clean URL.
The robots.txt rules for Meta's bots
To block meta-externalagent, add a robots.txt group naming it with Disallow: /. Meta's other four crawlers read their own groups, so blocking training leaves Meta AI citations alone:
User-agent: meta-externalagent
Disallow: /
To keep Meta AI able to cite you while blocking training, leave meta-webindexer allowed, or name it explicitly:
User-agent: meta-webindexer
Allow: /
Three things Meta's documentation adds:
- Changes take up to 24 hours. Meta asks you to allow that long for robots.txt edits to take effect.
- meta-externalfetcher may ignore robots.txt. Meta says that because it fetches links a user requested, it may bypass robots.txt rules. A page that must stay private needs authentication, not a Disallow line.
- facebookexternalhit may ignore it for security checks. Meta says the link-preview bot can bypass robots.txt when checking for malware or integrity problems. Blocking it also breaks the previews shown when people share your pages on Meta's apps.
Should you block meta-externalagent?
For most sites it isn't necessary. Blocking meta-externalagent leaves Meta AI's search citations alone, because those come from meta-webindexer, and it leaves every other engine alone too. What you give up is a place in whatever Meta trains next, and one site's logs can't measure what that costs.
The one Meta-specific reason to act is the sweep itself. 401 requests in a day is nothing for most sites, but a small server that slows under bursts can rate-limit the meta-externalagent user agent at the server or CDN rather than blocking it outright. Meta's documentation doesn't mention Crawl-delay, so don't rely on it.
To check whether Meta's other agents reach your site, GetIntel's AI traffic view splits requests by agent and day (Pro and Growth).
What our Meta logs could not see
Our Meta numbers are limited by IP verification, site type and sample length.
- Meta's other agents may be undercounted. Our DataFast setup keeps only requests it can verify against an operator's IP lists, and Meta's crawler page doesn't publish one, so we can't confirm which Meta ranges DataFast checks. meta-externalagent passed. Requests from the link-preview bot or the fetcher that failed verification would have been dropped, so a zero for those agents means none verified, not necessarily none sent.
- A B2B site whose links rarely get shared. A consumer brand whose links get shared on Facebook will see facebookexternalhit, and maybe meta-externalfetcher, where we saw none.
- Nine days is one data point. Meta publishes no crawl schedule, so we can't say whether this sweep comes monthly, quarterly or once.