CCBot: purpose, robots.txt rules and access checks
CCBot builds the Common Crawl web archive, which other organizations can use in datasets. Blocking future crawls does not remove old captures or directly control a search engine citation pipeline.
Aditya R. Singh maintains Let The Bots In, a technical AI-readiness scanner and reference site.
Technical reference
| Operator | Common Crawl |
|---|---|
| Class | Training / data-use control |
| HTTP User-Agent | CCBot/2.0 (https://commoncrawl.org/faq/) |
| Citation impact | Blocking future crawls does not remove old captures or directly control a search engine citation pipeline. |
| IP verification | Official IP ranges or verification reference |
Header versions can change. Use the product token for robots.txt matching and consult the Common Crawl crawler documentation before changing enforcement. Reference checked on 18 October 2026.
Common Crawl publishes an archive, not a live search product. This example allows other operators while blocking CCBot. Verify IPv4 reverse DNS under crawl.commoncrawl.org and resolve the hostname back to the original IP.
How do I allow CCBot?
Merge this group into the robots.txt served from the page's origin. Keep any path exclusions that still apply. A named group may stop inheriting the wildcard group's exclusions.
User-agent: CCBot
Allow: /Allow retrieval and block training
This example separates the operator's documented identities where possible. It changes crawl or data-use preferences; it does not grant server access. User-triggered agents may follow a different robots.txt policy.
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: CCBot
Disallow: /Use the interactive robots.txt builder for a per-bot policy. Test the final file against the exact page path before deploying it.
What if the CDN returns HTTP 403?
A robots.txt allow rule cannot override an HTTP 403, a login gate or a JavaScript challenge. Compare the returned body with the public page and locate the request in CDN security events. Grant an exception only after verifying the sender, scoped to the affected public routes.
The scanner uses its own network. A successful header probe does not verify Common Crawl access, and a refused probe does not establish a vendor-wide block. See the WAF troubleshooting guide.
Check your site's rules for CCBot
Frequently asked questions
What does CCBot do?
Common Crawl operates CCBot. CCBot builds the Common Crawl web archive, which other organizations can use in datasets.
Does blocking CCBot affect citations?
Blocking future crawls does not remove old captures or directly control a search engine citation pipeline.
How do I verify a CCBot request?
Use the operator's published verification reference linked on this page. Match the source IP, not just the User-Agent header. A scanner probe comes from its own network.
Check your site's technical readiness
Scan a public URL for crawler rules, delivered HTML and page metadata. The report distinguishes measured issues from checks it could not complete.
Check my site →