AI robots.txt checker

Check the robots.txt policy for a public URL against AI search and browsing identities. Find named-group blocks and wildcard rules, then build a policy that separates retrieval from training.

The scanner evaluates the deployed robots.txt against the exact page path. Use the browser-only draft tester for a file you have not deployed.

maintains Let The Bots In, a technical AI-readiness scanner and reference site.

Read the technical guide · Browse finding fixes

How do I run this check?

  1. Enter the public URL, including the path you want to inspect.
  2. Review the focused observations and the measurement coverage.
  3. Open a finding's fix guide, confirm the evidence and apply the relevant change.
  4. Rescan the same URL. Use Overview to inspect the full six-category breakdown.

Build a robots.txt policy

Allow retrieval while blocking training collection, or choose each bot yourself. Merge the output with your existing path restrictions. A named group may override exclusions in the wildcard group.

Search / retrieval
User-triggered browsing
Training / data-use control

Google-Extended and Applebot-Extended control data use rather than separate HTTP crawlers. Some user-triggered fetchers may ignore robots.txt. The file cannot protect private resources.

# Merge these groups with your existing path restrictions.
# robots.txt is public and does not protect private content.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Googlebot
Allow: /

User-agent: Google-Extended
Disallow: /

User-agent: bingbot
Allow: /

User-agent: CCBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Applebot
Allow: /

User-agent: Amzn-SearchBot
Allow: /

User-agent: Amzn-User
Allow: /

Frequently asked questions

Does Allow: / guarantee crawler access?

No. It permits crawling under robots.txt policy. HTTP blocks, authentication and CDN challenges can still prevent a usable response.

Can I block GPTBot while allowing ChatGPT search?

Yes. Use separate named groups for GPTBot and OAI-SearchBot. The builder below produces that separation and lets you review other operators.

Will a named allow group inherit wildcard exclusions?

Do not assume that it will. More-specific groups can replace the wildcard policy for a matching crawler. Preserve needed path restrictions in each applicable group.