--- title: "AI crawlers explained: who is fetching your site, and who to allow" slug: "ai-crawlers-explained" category: "geo" canonical_path: "/articles/geo/ai-crawlers-explained" meta_title: "AI crawlers: GPTBot, ClaudeBot & who to allow — Prime AI Visibility" meta_description: "What each AI crawler actually does — GPTBot, ClaudeBot, PerplexityBot — how to verify the real ones in your server logs, and a framework for deciding who to allow." author: "The Prime AI Visibility editorial team" date: "2026-07-10" last_updated: "2026-07-10" read_time: "11 min" keywords: - AI crawlers - GPTBot - ClaudeBot - PerplexityBot - robots.txt featured_image: "/brand/articles/geo/ai-crawlers-explained.png" featured_image_alt: "Three small geometric beetle shapes crossing a woven lattice toward two archways, one open and glowing, the other barred shut" og_image: "/brand/articles/geo/ai-crawlers-explained.og.png" cta_mid_headline: "Blocking or allowing — do you know what it changed?" cta_mid_body: "Prime AI Visibility tracks which answer engines cite your pages — and who they cite instead — across ChatGPT, Perplexity, Gemini, Claude, Copilot, and Google AI Overviews, every 24 hours." cta_mid_button: "See who cites you" cta_bottom_headline: "Crawl policy set? Now measure the outcome." cta_bottom_body: "robots.txt is an input. The output is whether engines actually name you. Create a workspace, add ten buyer prompts, and see your per-engine citations within 24 hours." cta_bottom_button: "Start tracking free" --- # AI crawlers explained: who is fetching your site, and who to allow AI crawlers are the automated agents that OpenAI, Anthropic, Perplexity, Google, and others send to fetch your pages — some to train models, some to build search indexes, some to retrieve a page live for one user's question. Each role has different GEO consequences, so a single blanket allow-or-block decision is almost always the wrong one. ## AI crawlers: the short answer 1. **There are three distinct jobs, not one.** Training crawlers collect content for future model weights, search-index crawlers feed answer-engine retrieval, and user-triggered fetchers grab a page live when someone asks about it. The same company usually runs all three under different user agents. 2. **Blocking is a per-job decision.** Blocking a training bot is a licensing stance with little visibility cost today. Blocking a search-index or user-triggered bot removes you from the answers where buyers would have seen you cited. 3. **User-agent strings are claims, not proof.** Anyone can send `GPTBot` in a header. The operators publish IP ranges and support reverse-DNS verification — your server logs, checked properly, are the only ground truth. ## The three jobs AI crawlers actually do Most of the confusion in crawl policy comes from treating "AI bots" as one category. The operators themselves split their fleets by purpose, and they document the split precisely so you can set different rules for each: - **Training crawlers** collect public web content that may end up in future model training runs. GPTBot and ClaudeBot are the canonical examples. What they fetch today may influence what a model "knows" in a release a year from now — a slow, compounding, unattributable effect. - **Search-index crawlers** build the retrieval indexes that answer engines query at answer time. OAI-SearchBot feeds ChatGPT search; Claude-SearchBot feeds Claude's web search; PerplexityBot feeds Perplexity's index. If these cannot reach you, the engine cannot retrieve you, and it certainly cannot cite you. - **User-triggered fetchers** act on behalf of one person, right now. ChatGPT-User, Claude-User, and Perplexity-User fetch a page because a live conversation needs it. This is the closest thing answer engines have to a human clicking your link. The distinction matters because the trade-offs run in opposite directions. Training access is a rights question with deferred, diffuse payoff. Retrieval and user-triggered access are visibility questions with immediate, measurable payoff — the pages those bots fetch are the pages that get quoted and cited in front of buyers. If you are optimizing for [share of citation](https://primeaivisibility.com/articles/geo/share-of-citation-explained), the second and third categories are where the metric is actually made. ## A field guide to the major AI crawlers Every entry below comes from the operator's own crawler documentation, linked in the References. All the listed bots identify themselves in the user-agent header and publish their robots.txt behavior. | User agent | Operator | Job | Respects robots.txt | |---|---|---|---| | `GPTBot` | OpenAI | Model training | Yes | | `OAI-SearchBot` | OpenAI | ChatGPT search index | Yes | | `ChatGPT-User` | OpenAI | User-triggered fetch | Yes | | `ClaudeBot` | Anthropic | Model training | Yes | | `Claude-SearchBot` | Anthropic | Claude search index | Yes | | `Claude-User` | Anthropic | User-triggered fetch | Yes | | `PerplexityBot` | Perplexity | Answer-engine index | Yes | | `Perplexity-User` | Perplexity | User-triggered fetch | Partial — see below | | `Google-Extended` | Google | Gemini training opt-out token | Yes (policy token) | | `CCBot` | Common Crawl | Open web archive (used in many training sets) | Yes | Operator-specific details worth knowing: - **OpenAI** documents all three roles at a single page and publishes machine-readable IP address lists for each. Blocking `GPTBot` in robots.txt does not block `OAI-SearchBot` or `ChatGPT-User` — the tokens are independent, which is exactly what makes a split policy possible. - **Anthropic** runs the same three-way split, plus a `claude-code` agent that fetches documentation on behalf of developers using its coding tool. Its training crawler, ClaudeBot, is also the most aggressive fetcher relative to the traffic it returns: Cloudflare's radar data has measured Anthropic's crawl-to-refer ratio in the tens of thousands of pages crawled per referral visit sent back. - **Perplexity** publishes IP lists for both of its bots. Its position on `Perplexity-User` is the contested one in the industry: Perplexity argues a fetch made at a specific user's request is not crawling, so that agent may fetch pages even where robots.txt disallows general crawling. Whether you accept the argument or not, plan your policy knowing it. - **Google** is the special case that breaks most people's mental model. `Google-Extended` is not a crawler you will see in logs — it is a robots.txt policy token, evaluated by ordinary Googlebot crawling, that controls whether your content trains and grounds Gemini. Blocking it does **not** remove you from Google AI Overviews, because AI Overviews is a search feature built on the ordinary search index. The only way to stay out of AI Overviews is to stay out of Google search — a trade almost nobody should make. - **Common Crawl's CCBot** is the quiet one that matters: its open archive has seeded many training datasets. Blocking CCBot is the broadest single training opt-out available, and it costs you nothing in answer-engine visibility because no engine retrieves from Common Crawl at answer time. ## Should you block AI crawlers? A decision framework There is no universal answer, but there is a clean way to decide. Ask three questions, in order: 1. **Does your business earn from being found, or from the content itself?** If content *is* the product — paywalled journalism, licensed data, subscription research — training bots are extracting your asset, and blocking GPTBot, ClaudeBot, and CCBot is a defensible licensing position. If content exists to make buyers discover and trust you — most B2B sites, most product companies — the extraction concern is small and the discovery upside is the entire point. 2. **Which surfaces do your buyers actually use?** If your prompts show up in ChatGPT and Perplexity, blocking OAI-SearchBot or PerplexityBot is self-sabotage: the engine cannot cite a page it cannot retrieve. We have [compared how the two engines behave as citation surfaces](https://primeaivisibility.com/articles/geo/perplexity-vs-google-ai-overviews) — they differ in freshness and source count, but both require crawl access to name you. 3. **Can you measure the consequence?** Any crawl-policy change is a hypothesis about visibility. If you cannot see your per-engine citation rate before and after the change, you are flying blind. This is a solvable problem — [tools that track AI search citations](https://primeaivisibility.com/compare/ai-search-trackers) exist precisely to close that loop. The pattern that falls out for most commercial sites: - **Allow** all search-index and user-triggered bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User). These are the paths to being cited. - **Decide deliberately** on training bots (GPTBot, ClaudeBot, CCBot, Google-Extended) based on question one. Reasonable companies land on both sides. - **Never** block by IP reputation lists or generic bot protection without checking what is in them — over-aggressive bot mitigation silently blocks the AI crawlers you meant to welcome, and nothing in your analytics will tell you. One caution on rate limiting rather than blocking: training crawlers can be heavy. Cloudflare's published data shows AI crawl volume growing while referral clicks shrink, which is exactly why the crawl-to-refer ratio has become the industry's shorthand for the imbalance. If server load is the concern, rate-limit the training bots and leave the retrieval bots alone — do not solve a bandwidth problem by deleting yourself from answers. ## Verify who is actually hitting your server User-agent strings are self-reported, and the famous ones are the most-spoofed — scrapers borrow the `GPTBot` label to inherit its welcome. Before you conclude anything from your logs, verify: 1. **Check the operator's published IP list.** OpenAI, Anthropic, and Perplexity all publish machine-readable IP ranges for each bot. A request claiming to be PerplexityBot from an IP outside `perplexitybot.json` is an impostor. 2. **Use forward-confirmed reverse DNS where supported.** Reverse-resolve the source IP, confirm the hostname belongs to the operator's domain, then forward-resolve that hostname and confirm it returns the original IP: ```bash dig -x 20.171.207.1 # → something like crawl-xx.openai.com dig crawl-xx.openai.com # → must return 20.171.207.1 ``` 3. **Then read the story your logs tell.** Which bots fetch you, how often, and which pages? A healthy GEO posture shows search-index bots hitting your key commercial pages and user-triggered fetchers arriving when buyers ask about your category. If the only AI traffic you see is training crawlers bulk-reading your blog, engines are learning *from* you without ever *citing* you — the worst trade available. This is the same log discipline that tells you whether anyone reads your llms.txt — and it is worth being clear about [what llms.txt does and does not control](https://primeaivisibility.com/articles/geo/what-is-llms-txt): it is a curation hint, not an access control. Crawl permissions live in robots.txt and nowhere else. ## Misconceptions that cost real visibility - **"Blocking AI crawlers protects my SEO."** Access and ranking are different layers. Blocking GPTBot changes nothing about Google rankings; blocking Googlebot to escape AI Overviews destroys your search presence entirely. Know which layer each token controls before you edit robots.txt. - **"I blocked training bots, so my content can't appear in AI answers."** False. Answers are built from retrieval at question time. If OAI-SearchBot and PerplexityBot can reach you, engines can quote and cite you regardless of your GPTBot rule — and models may already carry older copies of your pages from before the block, or from Common Crawl if CCBot was ever allowed. - **"AI crawlers ignore robots.txt anyway, so why bother."** The documented major bots — GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot — honor it, and the operators say so in writing. The genuine gray zone is narrow: user-triggered fetchers, where Perplexity openly argues robots.txt does not apply to on-demand retrieval. Do not let a narrow dispute justify abandoning the one control the entire fleet respects. - **"More crawling means more visibility."** Volume is not value. Crawl-to-refer data shows training bots reading thousands of pages for every visit any AI platform sends back. The number that matters is not how often AI crawlers fetch you — it is how often engines cite you when your buyers ask. ## What your crawl policy means for GEO Crawl access is the zeroth step of generative engine optimization: necessary, never sufficient. An allowed bot still has to find pages worth retrieving, and a retrieved page still has to win the citation against every competitor the engine also retrieved. That is a content problem — direct answers, sourced claims, stable headings — and a measurement problem: knowing, per engine, whether you or your competitor got named. So treat robots.txt as the cheap, reversible input it is. Set the policy deliberately using the framework above, verify the real AI crawlers in your logs quarterly, and put your ongoing attention where the compounding returns are — publishing material engines want to quote, and measuring whether they do. > **Blocking or allowing — do you know what it changed?** > > Prime AI Visibility tracks which answer engines cite your pages — and who they cite instead — across ChatGPT, Perplexity, Gemini, Claude, Copilot, and Google AI Overviews, every 24 hours. > > **[See who cites you](https://app.primeaivisibility.com/sign-up)** ## References 1. OpenAI, *Overview of OpenAI crawlers and user agents* (GPTBot, OAI-SearchBot, ChatGPT-User). 2. Anthropic, *Does Anthropic crawl data from the web?* (ClaudeBot, Claude-SearchBot, Claude-User). 3. Perplexity AI, *Perplexity crawlers* (PerplexityBot, Perplexity-User, published IP lists). 4. Google Search Central, *Google's common crawlers and user agents* (Google-Extended). 5. Cloudflare, *The crawl before the fall of referrals: understanding AI's impact on content providers* (crawl-to-refer ratios). 6. Cloudflare, *A deeper look at AI crawlers: breaking down traffic by purpose and industry*. ## Next steps 1. **[See the use cases teams track with citation data](https://primeaivisibility.com/use-cases)** — what marketing, product, and comms teams actually do with per-engine visibility. 2. **[Read how content wins the citation once the bot arrives](https://primeaivisibility.com/articles/geo/how-to-write-content-chatgpt-will-quote)** — crawl access gets you retrieved; the writing gets you quoted. 3. Before you change a single robots.txt line, **[create a Prime AI Visibility workspace](https://app.primeaivisibility.com/sign-up)**, add ten buyer prompts, and baseline your citations so you can see what the change does. ## Frequently asked questions **What are AI crawlers?** AI crawlers are automated agents operated by AI companies that fetch web pages for three distinct purposes: training future models (GPTBot, ClaudeBot), building the search indexes answer engines retrieve from (OAI-SearchBot, PerplexityBot), and fetching pages live on behalf of a user mid-conversation (ChatGPT-User, Claude-User, Perplexity-User). Each identifies itself with a documented user-agent string. **Should I block GPTBot?** Only if content is your product and you object to it training OpenAI's models — that is a legitimate licensing stance. For most commercial sites, the calculus favors allowing it, and either way the decision does not affect whether ChatGPT search can cite you: that depends on OAI-SearchBot, a separate token. **Does blocking Google-Extended remove me from AI Overviews?** No. Google-Extended only controls whether your content trains and grounds Gemini models. AI Overviews is built on Google's ordinary search index via Googlebot, so the only way out of AI Overviews is out of Google search itself — which is almost never worth it. **How do I know if an AI crawler is real or spoofed?** Match the source IP against the operator's published IP list — OpenAI, Anthropic, and Perplexity all publish them — and use forward-confirmed reverse DNS: reverse-resolve the IP, check the hostname belongs to the operator, forward-resolve it back to the same IP. A user-agent string alone proves nothing. **Do AI crawlers respect robots.txt?** The major documented bots do, and their operators commit to it in writing. The one open dispute is user-triggered fetching: Perplexity argues that a page fetched at a specific user's request is not crawling, so Perplexity-User may fetch where general crawling is disallowed. Set your policy with that boundary in mind. **Which AI crawlers should a typical B2B site allow?** Allow every search-index and user-triggered bot — OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User — because they are the path to being cited in buyer conversations. Make a deliberate, separate call on the training bots (GPTBot, ClaudeBot, CCBot, Google-Extended) based on whether your content is the product or the marketing. > **Crawl policy set? Now measure the outcome.** > > robots.txt is an input. The output is whether engines actually name you. Create a workspace, add ten buyer prompts, and see your per-engine citations within 24 hours. > > **[Start tracking free](https://app.primeaivisibility.com/sign-up)**