Compare AI crawler identities, robots.txt controls and the effect of blocking each bot. Search crawlers, training collection and user-requested fetches need separate rules.
Match the product token in robots.txt. Verify source IPs before granting WAF exceptions. A copied User-Agent header proves nothing about who sent a request.
| Bot and operator | Purpose | What blocking changes |
|---|---|---|
| OAI-SearchBotOpenAI | Search / retrieval | Opting out excludes content from ChatGPT search answers; navigational links can still appear. |
| GPTBotOpenAI | Training / data-use control | Blocking GPTBot expresses a training opt-out. ChatGPT search has its own OAI-SearchBot control. |
| ChatGPT-UserOpenAI | User-triggered browsing | A server block can prevent an on-demand fetch. This agent does not control ChatGPT search inclusion. |
| Claude-SearchBotAnthropic | Search / retrieval | Disallowing it prevents this indexing path and may reduce search visibility. Access alone does not establish citation eligibility. |
| ClaudeBotAnthropic | Training / data-use control | Blocking it limits this training crawl; Claude search and user fetches have separate controls. |
| Claude-UserAnthropic | User-triggered browsing | Blocking it can prevent Claude from retrieving the requested page through this agent. |
| PerplexityBotPerplexity | Search / retrieval | Blocking it limits search discovery through this crawler. It is separate from the user-requested fetcher. |
| Perplexity-UserPerplexity | User-triggered browsing | A server block can stop that fetch, including its use as a linked source in the answer. |
| GooglebotGoogle | Search / retrieval | Blocking crawling can prevent Google from fetching new content. Indexing and snippet eligibility are separate checks. |
| Google-ExtendedGoogle | Training / data-use control | This control does not affect Google Search inclusion or ranking. Gemini Apps grounding is a separate use covered by this token. |
| bingbotMicrosoft | Search / retrieval | Blocking it can limit discovery in Bing and products that retrieve through that index, including Copilot. An allow rule promises no citation. |
| CCBotCommon Crawl | Training / data-use control | Blocking future crawls does not remove old captures or directly control a search engine citation pipeline. |
| Applebot-ExtendedApple | Training / data-use control | Blocking this token does not exclude pages from Apple search results. |
| AmazonbotAmazon | Training / data-use control | Blocking Amazonbot opts out of this collection. Amazon now documents separate search and user fetchers. |
| ApplebotApple | Search / retrieval | Blocking Applebot can limit retrieval through Apple search. Applebot-Extended controls training use separately. |
| Amzn-SearchBotAmazon | Search / retrieval | Allowing this crawler supports search discovery without allowing Amazonbot training collection. |
| Amzn-UserAmazon | User-triggered browsing | Server access affects this on-demand retrieval. It has a separate identity from Amazonbot. |
Build and check robots.txt to allow search while blocking training. Use the WAF response tester when the file allows a bot but requests still fail.
Google-Extended and Applebot-Extended are data-use controls with no separate crawling header. Amazon now separates Amazonbot collection from Amzn-SearchBot retrieval.
Aditya R. Singh maintains Let The Bots In, a technical AI-readiness scanner and reference site.