How this tool works

The pipeline

A scan makes six network requests, all in parallel, none allowed to block the others. The whole thing runs against public data only: no login, no JavaScript execution, no browser.

Site-level files (robots.txt, sitemap.xml, llms.txt) are always fetched from the domain's origin. Page-level analysis (structured data, headings, readability) runs against the exact URL you submitted, so scanning an article URL doesn't silently fall back to homepage signals.

The three-class crawler model

Every AI crawler falls into one of three intents, and the intent determines what blocking it actually costs you. This is the distinction most robots.txt advice misses: "block AI bots" is not one decision, it's at least three.

Training crawlers feed model pre-training. Blocking them is a legitimate IP or privacy decision and this tool does not penalize it. Search / retrieval crawlers power live citations. Blocking one of these is the single highest-severity finding the scan can produce. User-triggered crawlers fetch a page only when a person explicitly asks an assistant to open or summarize that URL; blocking them doesn't reduce your general visibility, but it does mean a user pasting your link into ChatGPT or Claude gets a dead end. A few bots (noted individually) assert they don't honor robots.txt at all, so the scan flags those separately rather than reporting a "blocked" verdict robots.txt can't actually enforce.

robots.txt parsing

Rules are grouped by User-agent: line. A crawler's own named group (e.g. User-agent: OAI-SearchBot) always overrides a wildcard User-agent: * group, even if the wildcard group appears first in the file. Within whichever group applies, the longest matching path wins: not rule order, not Allow-before-Disallow. That single rule is why "Disallow everything, then Allow a subfolder" reliably does what site owners intend, and why forgetting it produces false "blocked" reports.

Both rules match the request path. robots.txt precedence goes to the longer matching path, not the order rules appear in, so the nested Allow wins even though Disallow: /blog/ comes first in the file.

The scan also flags patterns beyond a simple allow/block verdict: a wildcard Disallow: / that blocks every AI search bot at once (the signature of an SEO plugin's "block AI crawlers" toggle left on by default), robots.txt that explicitly names training bots but has no rules at all for search/retrieval bots (the most common silent loss: nothing is technically "blocked," the site owner just never opted in), and Crawl-delay values of an hour or more aimed at AI bots specifically, which throttle indexing as effectively as a block would.

Access

The gate. If a crawler can't reach your content, nothing else on this page matters, which is why Access is worth more than any other category.

A missing robots.txt (non-200 response) is treated as "everything allowed by default," per the robots.txt spec, but the scan reports it as a hygiene gap regardless, since having no explicit rules also means you have no way to opt specific bots in or out later.

If the homepage request itself is refused (401/403/429/451, or a 200 that's actually a bot-verification challenge page), none of the above runs: there's no score, and no claim about your H1, meta description, or structured data, because we never received the HTML to check it in. See "We are not a verified crawler" below for why that refusal doesn't mean AI crawlers can't reach you either — and Common Crawl for how we recover a content-only read in that case.

Readability

Being "allowed" doesn't help if there's nothing to read. AI crawlers fetch raw HTML. They do not run your JavaScript bundle, so anything that only exists after client-side rendering is invisible to them, full stop.

"JS-only" is detected by two signals together: a known SPA root element (#root, #app, #__next, #__nuxt) and fewer than 200 characters of visible text in the raw HTML body. Either signal alone isn't enough: plenty of non-SPA pages happen to have a div id="app" somewhere, and plenty of legitimately short pages aren't JS shells. Paywall detection looks for common phrasing ("subscribe to read," "members only," "sign in to view") rather than a specific paywall vendor, so it catches custom implementations too. A login gate is a password input combined with under 600 characters of visible content, on the assumption that if there's a login form and almost nothing else, the real content is behind it.

Structured data

Structured data is how AI systems identify what your site is and who runs it with confidence, instead of guessing from prose. JSON-LD is the only format that earns meaningful points. It's the format every major AI vendor's documentation recommends, because it's unambiguous and separated from display markup.

Type points stack. A page with Organization, WebSite, and FAQPage schema earns all three. There's no cap below the category max, but the six-plus types above already sum past 16, so realistic coverage tops out at the category ceiling regardless.

Authority & entity

This is the category AI systems weigh most heavily when deciding whether to trust and attribute a source. Structured data tells a crawler what a page says; authority signals tell it who's saying it.

The Wikidata check is the one live third-party lookup in the whole pipeline. Everything else is derived purely from what your own site returns. It runs with its own 3-second budget and never blocks or fails the rest of the scan if it times out.

Extractability

Reachable, readable, and identifiable still isn't the same as quotable. This category measures whether your content is structured the way AI systems actually lift text from. A page can pass every check above and still be one giant unbroken paragraph that's hard to extract a clean answer from.

Freshness & hygiene

The smallest category, deliberately: these are real signals, but none of them individually determine whether AI engines can find or cite you the way Access does.

llms.txt is weighted deliberately low. There's little public evidence today that any major AI vendor's retrieval pipeline reads it, and no vendor has committed to supporting it as a ranking or citation input, so this tool reports it as hygiene, not as something that will move your citation rate. If that changes, the weight changes; nothing on this page or the score card overstates it in the meantime.

Score composition

The six category scores sum directly to your 0–100 total. There's no additional weighting or normalization step.

Bands

Here's one finding traced end to end, from what the scanner detected to the fix it generated:

Bot view vs. browser view

Every scan fetches the same URL twice, once with a normal browser user-agent and once with an AI crawler's user-agent, and strips both responses down to plain text the same way an extractor would (scripts, styles, comments, and markup removed; block-level tags become line breaks so headings and paragraphs stay separable).

This catches two distinct problems. A large gap between the two word counts, when the browser count is substantial, flags user-agent-based cloaking: a CDN or WAF rule serving different content by user-agent, intentionally or not. A very low bot-view word count regardless of the browser count flags JS-only rendering: there was never a cloaking rule, the content genuinely doesn't exist until the browser runs your JavaScript.

Common Crawl

Common Crawl is a public web archive that a large share of LLM training pipelines draw from, directly or indirectly. This section is historical context, not a live citation signal. It tells you whether CCBot was allowed to crawl you in the past, and whether it actually captured your page when it tried.

Bar color = whether CCBot was allowed in that crawl's robots.txt. Bar height = whether the page was actually captured. The dashed bar is a live probe run right now, not archived history. CCBot can be re-allowed today even after months of being blocked.

The scan also runs a live probe against your current robots.txt (using CCBot's user-agent, not its IP; a UA-only check can't perfectly replicate what CCBot's actual crawler sees, so this is reported as indicative, not definitive) and, where detectable, names the likely source of a block: certain hosting platforms and CDNs ship a managed robots.txt template that blocks CCBot by default.

When our own homepage request is refused (see Access), we separately check whether Common Crawl already holds a capture of the page — CCBot is a verified crawler at most WAFs, so it's often let through where we aren't. If one exists, we run the same content analysis (structured data, headings, meta description, author signals) against that archived HTML and show it as its own section, dated to the capture, with no score attached. Common Crawl stores raw HTML, which is exactly the non-JavaScript view a real AI crawler gets, so this isn't a workaround for the refusal, it's the same underlying question — "is this page structured for AI citation" — answered from a copy we didn't have to fetch ourselves. It says nothing about whether OAI-SearchBot or ClaudeBot can reach you today; that's still unknown, and stays unknown.

Limits

A methodology page that doesn't say what it can't do is marketing, not methodology. Here's what this tool structurally cannot tell you:

• It doesn't measure actual AI citations. Nothing here confirms ChatGPT, Claude, or Perplexity have ever cited you, or will. It checks the technical preconditions (reachability, readability, structure, identity) that citation depends on, not the outcome itself.
• It checks one page, not your whole site. The homepage (or the specific URL you paste) plus robots.txt, sitemap.xml, and llms.txt at the domain root. A deep article three clicks in might have completely different structured data or heading structure, and this scan won't see it.
• It never executes JavaScript. Every fetch is a raw HTTP request read as text. If your content requires client-side rendering to appear, the scan sees exactly what an AI crawler sees: nothing, or close to it. That's the point, but it also means the scan can't distinguish "broken" from "renders fine in a real browser, just not here."
• Some bots ignore robots.txt regardless of what it says. Where a crawler is known to not honor robots.txt rules (Perplexity-User is the current example, per its own published documentation), the scan reports that explicitly instead of a misleading "blocked" verdict robots.txt can't actually enforce.
• llms.txt is treated as hygiene, not a citation driver. See the Freshness & hygiene section above for why.
• We are not a verified crawler, and can't offer IP allowlisting. This scan runs on Cloudflare Workers, sending requests as LetTheBotsIn-Checker/1.0 (+https://letthebots.in/about) from Cloudflare's large, shared, dynamic edge IP pool, not a small fixed range. Enterprise bot managers (Akamai Bot Manager, Cloudflare Bot Management, Imperva, DataDome) verify real AI crawlers by source IP against each vendor's published ranges, and treat everything else, us included, as unverified traffic. So when your site refuses us, that refusal is evidence about our standing with your WAF, not about whether OAI-SearchBot, ClaudeBot, or PerplexityBot get through. The scan reports that distinction explicitly rather than scoring your site down for it, and the fix it suggests is to check your own server or CDN logs for those crawlers' published user-agents, since that's the only vantage point that can actually answer the question.

Stack

Astro, deployed on Cloudflare Workers. Stateless by default: a scan is computed fresh on every request and nothing about your site is stored. The two exceptions are an email address if you explicitly ask to be monitored for regressions, and a cached Common Crawl API response (shared across users, keyed by domain, to avoid re-querying Common Crawl's own rate-limited API on every visit). Both live in Cloudflare KV.