Fix HTTP 403s for AI crawlers behind Cloudflare and other WAFs

Is the crawler disallowed or refused?

Fetch the exact URL that failed. Keep the scheme, hostname and path. A policy on example.com does not describe www.example.com, and an accessible homepage does not establish that /docs/ works.

robots.txt expresses crawl preferences. The CDN and origin decide whether a request receives an article, a 403, a login page or a challenge. Diagnose those layers separately. Otherwise you can spend an afternoon editing a file that the blocking rule never reads.

curl -sS -D robots.headers -o robots.txt 'https://example.com/robots.txt'
curl -sS -D browser.headers -o browser.html 'https://example.com/docs/'
curl -sS -A 'OAI-SearchBot' -D bot.headers -o bot.html 'https://example.com/docs/'

• Run the browser and bot-header comparison
• OAI-SearchBot identity and verification

Read the body as well as the status

A 403 identifies a refusal for that request. It does not tell you whether the cause is a custom rule, a managed rule, rate limiting or the application. A 200 can also contain a challenge page instead of useful text. Read the response before calling it a pass.

The command above copies a bot header. It does not turn your terminal into OpenAI infrastructure. Bot managers can make decisions using source addresses and other request properties. Treat a header-dependent difference as a lead to investigate in security events, not proof that all ChatGPT retrieval is blocked.

1. Compare the response status, content type, redirect location and main text for both requests.
2. Find a request at the same timestamp and path in your CDN events. Record the action and rule ID.
3. Inspect access logs for real OAI-SearchBot or PerplexityBot requests and verify their source IPs against the operator reference.

• PerplexityBot IP reference
• Inspect crawler access logs locally

Scope the exception to the false positive

Cloudflare documents cf.client.bot as a known-bot signal. The example below narrows a candidate match to public documentation GET requests. Use it only after confirming that the affected crawler is recognized in your events and that these routes should be accessible.

Choose Skip for the specific security checks that caused the false positive, and place the rule where it can affect those checks. Do not exempt every request with an AI-looking User-Agent. Any client can copy that string. Keep login routes and unrelated security protections outside the exception.

cf.client.bot
and http.request.method eq "GET"
and starts_with(http.request.uri.path, "/docs/")

• Cloudflare known-bot rule reference
• Cloudflare Skip action and available options

Check managed bot settings before adding another rule

A CDN can prepend managed rules to robots.txt or block an AI category through a dashboard setting. Compare the file served over HTTP with the one in your repository. If they differ, find the setting that owns the extra lines.

Cloudflare Bot Fight Mode cannot be bypassed with a custom-rule Skip action. That limit matters when a carefully written exception seems to have no effect. Confirm which product generated the event instead of adding broader exceptions. Product names and dashboard locations can change; follow the current rule reference linked above.

• Inspect a managed robots.txt section
• Interpret the scanner block finding

What counts as a successful fix?

Request the page again and confirm that the intended public text arrives. Then look for later verified vendor requests with usable responses. Keep the timestamp, source verification and rule change together so you can explain why the behavior changed.

A successful fetch establishes delivery at that moment. Index updates and citation selection happen later and are outside this test. If the scanner remains blocked while verified vendor traffic receives the article, document that distinction rather than weakening security to improve a diagnostic score.