Resolve an uncertain server comparison

What does this finding mean?

The report identifies this check as server-block-uncertain in Access. The category carries up to 28 of the 100 rubric points. This is a category weight, not an automatic deduction for this finding. Read the scan detail for the measured state, overlapping signals and any score cap.

Retry the same URL and inspect logs. Keep unknown access separate from a confirmed denial.

Why does this matter for AI retrieval?

A retrieval system needs a usable HTTP response before it can read the page. robots.txt policy and server enforcement are separate observations. ChatGPT, Claude and Perplexity have distinct crawler controls; Google Search AI features depend on Google crawling and indexing.

How do I check and resolve it?

1. Compare the browser and bot-header status, redirect chain, content type and body for the same URL. A 200 challenge page is still not the article.
2. Find the request in CDN security events. Record the rule ID, action, source IP and path. Check genuine crawler source addresses against the operator reference before granting an exception.
3. Scope an exception to the public routes and the rule that caused the false positive. A User-Agent header alone is forgeable. Keep authentication and unrelated security checks.

Adapt the example to the reported issue and the actual route. Placeholder names, URLs and dates must be replaced with facts from your site.

Copy code

curl -sS -D browser.headers -o browser.html 'https://example.com/page/'
curl -sS -A 'OAI-SearchBot' -D bot.headers -o bot.html 'https://example.com/page/'
# Candidate Cloudflare match, scoped to public GETs and verified bots:
# cf.client.bot and http.request.method eq "GET"
# and starts_with(http.request.uri.path, "/learn/")

Verify the result

Inspect subsequent verified crawler requests in logs. A scanner request with a bot header comes from our network, so a refusal cannot establish that the vendor is blocked.

Open the related tool or technical guide, then run a new scan of the affected URL. Keep unknown checks marked unknown until you have a complete response.

Copy a prompt for your coding agent

Paste this into Codex, Claude Code or Cursor from your own repository. Include the scan's exact URL and detail so the agent can verify the observation.

Copy prompt for coding agent

Investigate the scanner finding "Resolve an uncertain server comparison" (server-block-uncertain) in this repository.
Observation: The limited request comparison did not establish a reliable crawler-specific block.
Required change: Retry the same URL and inspect logs. Keep unknown access separate from a confirmed denial.
Inspect the existing route, shared layouts and configuration first. Confirm the finding against the actual response before editing. Preserve intentional access restrictions, existing content and unrelated changes. Use only factual identity and date values. Make the smallest appropriate fix, verify the deployed HTML or resource response and run the relevant project checks. Explain any measurement that remains unknown. Do not promise search rankings or AI citations.
Reference: https://letthebots.in/fixes/server-block-uncertain/