Verify permitted search crawler rules

What does this finding mean?

The report identifies this check as search-bots-accessible in Access. The category carries up to 28 of the 100 rubric points. This is a category weight, not an automatic deduction for this finding. Read the scan detail for the measured state, overlapping signals and any score cap.

Keep the policy if intended. Inspect server logs separately because a rule cannot prove HTTP access or citations.

Why does this matter for AI retrieval?

A retrieval system needs a usable HTTP response before it can read the page. robots.txt policy and server enforcement are separate observations. ChatGPT, Claude and Perplexity have distinct crawler controls; Google Search AI features depend on Google crawling and indexing.

How do I check and resolve it?

1. Fetch /robots.txt from the exact origin, including the www or subdomain used by the page. Save the current file before editing.
2. Find the group that matches the affected bot. More-specific groups can replace wildcard rules; copy any path exclusions you still need into the named group.
3. Merge the relevant example groups into the existing file. Replace example.com and keep private resources behind authentication.

Adapt the example to the reported issue and the actual route. Placeholder names, URLs and dates must be replaced with facts from your site.

Copy code

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

Verify the result

Fetch the deployed file with curl, then test the exact page path in the robots.txt tester. A permitted rule says nothing about the server response to a verified crawler.

Open the related tool or technical guide, then run a new scan of the affected URL. Keep unknown checks marked unknown until you have a complete response.

Copy a prompt for your coding agent

Paste this into Codex, Claude Code or Cursor from your own repository. Include the scan's exact URL and detail so the agent can verify the observation.

Copy prompt for coding agent

Investigate the scanner finding "Verify permitted search crawler rules" (search-bots-accessible) in this repository.
Observation: The parsed robots.txt policy permits the checked search crawlers at this path.
Required change: Keep the policy if intended. Inspect server logs separately because a rule cannot prove HTTP access or citations.
Inspect the existing route, shared layouts and configuration first. Confirm the finding against the actual response before editing. Preserve intentional access restrictions, existing content and unrelated changes. Use only factual identity and date values. Make the smallest appropriate fix, verify the deployed HTML or resource response and run the relevant project checks. Explain any measurement that remains unknown. Do not promise search rankings or AI citations.
Reference: https://letthebots.in/fixes/search-bots-accessible/