Public API and CI check

Everything the site does is available as plain HTTP. No key, no account. Please keep it to what you need: the scan endpoint allows 10 requests every 60 seconds per IP address.

Scan a URL

GET https://letthebots.in/api/scan?url=example.com/pricing

url can be a bare domain or a full URL. Site-level files (robots.txt, sitemap, llms.txt) always come from the origin; page-level checks run against the exact scheme, path and query string. Responses are never cached, and CORS is open, so you can call it from a browser.

curl -s "https://letthebots.in/api/scan?url=example.com" | jq '{score, band, blocked: [.platforms[] | select(.status=="blocked") | .name]}'

Response fields

Field	Type	Meaning
score	number \| null	0–100 for complete measurements, or null for unscoreable or partial scans. Null is not zero.
scoringVersion	string	Version of the readiness rubric. Interpret score differences only when versions match.
coverage	object	Measurement coverage, including complete and per-check observations. Unknown checks are not passes.
band	string \| null	invisible, at-risk, solid or citation-ready (legacy key; displayed as Strong technical readiness).
scoreCap	object \| null	Set when an observed policy or response check limits the score: {id, cap, reason}.
pageType	string	What we think the page is (home, article, docs, collection…). Drives schema advice.
pageFetch	object	{status: ok \| blocked \| error, httpStatus, reason}.
platforms	array	Configured crawler policy summaries. Status may be allowed, blocked, partial, ignores-robots or unknown. This is not evidence of vendor visits or citations.
robotsTxt	object	Includes found, snippet and state: detected \| not-detected \| could-not-check. A failed request is not proof of absent rules.
crawlers	array	Every crawler we test: {ua, operator, class, status}.
scoring	object \| null	Per-category {score, max, label}. Inspect coverage before interpreting these measurements as complete.
findings	array	{id, category, severity, title, detail, fix}. Severity is critical, high, medium, low, info or pass.
fixes	object	Generated robots.txt, JSON-LD and llms.txt snippets, when relevant.
agentSignals	object \| null	Markdown negotiation, llms-full.txt, Content-Signal, snippet controls.
botView	object \| null	Text extracted from the observed response. A browser rendering, if requested and available, is a separate optional check.

Errors come back as {"error": "..."} with status 400 (bad input), 429 (rate limited, see Retry-After) or 5xx. A site that refuses our request can return a successful API response with score: null and pageFetch.status: "blocked". Timeouts, incomplete HTML and failed resources can also leave a scan incomplete. Read coverage and the individual observations before treating a missing measurement as a problem with the site.

Current scoring version: 2026-10-18.3. Keep scoringVersion with exported results and monitor baselines. Rebaseline after a version change; a numerical drop across versions does not establish a regression. Requests with unknown robots policy must not be treated as confirmed crawler blocks.

Fail a deploy that blocks AI crawlers

Paste this into .github/workflows/ai-readiness.yml. It fails when a required configured crawler policy is disallowed, when our user-agent probes are refused, when the measurement is incomplete, or when a complete score falls below your threshold, and writes a summary to the run page. It only needs curl and jq, so the same script works in GitLab CI, CircleCI or a cron job.

name: AI crawler access
on:
  push:
    branches: [main]
  schedule:
    - cron: '0 6 * * *'   # daily, catches CDN or plugin changes too

jobs:
  ai-readiness:
    runs-on: ubuntu-latest
    steps:
      - name: Check AI readiness
        env:
          SITE: example.com            # your domain (add a path to test one page)
          MIN_SCORE: 65                # fail below this score
          REQUIRE: ChatGPT,Claude,Perplexity,Gemini,Copilot   # must not be blocked
        run: |
          set -euo pipefail
          res=$(curl -fsS --retry 2 --max-time 60 \
            "https://letthebots.in/api/scan?url=$(jq -rn --arg s "$SITE" '$s|@uri')")
          score=$(jq -r '.score // "none"' <<<"$res")
          band=$(jq -r '.band // "unreachable"' <<<"$res")
          blocked=$(jq -r --arg req "$REQUIRE" \
            '[.platforms[] | select(.status=="blocked") | .name] as $b
             | ($req | split(",")) | map(select(. as $p | $b | index($p))) | join(", ")' <<<"$res")
          server=$(jq -r '[.findings[] | select(.id=="server-block-detected")] | length' <<<"$res")

          {
            echo "### AI readiness for $SITE: $score/100 ($band)"
            jq -r '.findings[] | select(.severity=="critical" or .severity=="high") | "- **\(.severity)**: \(.title)"' <<<"$res"
            echo; echo "[Full report](https://letthebots.in/r/$SITE)"
          } >> "$GITHUB_STEP_SUMMARY"

          fail=0
          if [ "$score" = "none" ]; then echo "::error::No complete score; inspect pageFetch and coverage"; fail=1; fi
          if [ -n "$blocked" ]; then echo "::error::Disallowed configured crawler policies: $blocked"; fail=1; fi
          if [ "$server" != "0" ]; then echo "::error::Server refuses AI crawler user-agents"; fail=1; fi
          if [ "$score" != "none" ] && [ "$score" -lt "$MIN_SCORE" ]; then
            echo "::error::Score $score is below $MIN_SCORE"; fail=1
          fi
          exit $fail

Want alerts rather than failed builds? Scan your site and use Monitor this site at the bottom of the report: scheduled checks, email plus Slack, Discord or webhook alerts. Uncertain access failures require two consecutive scheduled observations; a scoring-version change does not produce a score-drop alert.

Badge

https://letthebots.in/badge/example.com.svg

An SVG badge with the latest score for a domain (the score from its most recent scan in the last 30 days). Unscanned domains show "not scanned". Reports in the highest readiness band show ready-made HTML and Markdown snippets. A badge does not establish vendor access or predict citations.

Share image

https://letthebots.in/og/example.com.png?p=/optional/path

A 1200×630 PNG of the latest score, used automatically as the social preview for /r/ report links.

Raw robots.txt

GET https://letthebots.in/api/robots?url=example.com

Returns {domain, status, found, content}. Used by the robots.txt tester.

Published crawler IP ranges

GET https://letthebots.in/api/bot-ranges

The IP ranges OpenAI, Anthropic, Google, Microsoft, Perplexity, Apple, DuckDuckGo and Common Crawl publish for their crawlers, merged and refreshed daily. Used by the log analyzer.