WordPress AI visibility: schema, bot blocks and llms.txt
Check WordPress search visibility, the public HTML response and crawler blocks before adding more plugins. Keep one source of publisher schema, inspect security-plugin decisions and treat llms.txt as an optional documentation index.
Aditya R. Singh maintains Let The Bots In, a technical AI-readiness scanner and reference site.
Is the public site asking search engines to stay out?
In Settings, Reading, check the search-engine visibility setting. WordPress can emit a noindex instruction when the site is configured to discourage indexing. That is useful on staging and easy to forget after launch. Change it only on a site meant for public search.
Fetch a published article while signed out and inspect response headers as well as the head element. A plugin or hosting rule can add X-Robots-Tag even when the WordPress setting looks right. Do not use search visibility as access protection for private content; require authentication.
curl -sS -D page.headers -o page.html 'https://example.com/a-real-post/'
rg -ni 'x-robots-tag|location:' page.headers
rg -ni 'noindex|nosnippet|canonical|application/ld\+json' page.htmlFind the rule that blocked the request
A security plugin, hosting firewall and CDN can all act on the same request. Compare a normal fetch with a crawler-header probe, then locate the matching event at the layer that refused it. A generic 403 template is not enough to identify the owner.
Keep admin, login and account routes protected. For public posts, scope a verified-crawler exception to the specific false-positive rule. Never grant privileged access to a request solely because it calls itself GPTBot or OAI-SearchBot.
- Fetch /robots.txt from the production hostname and compare it with the policy you intended.
- Check security-plugin logs, host access logs and CDN events at the same timestamp.
- Verify actual crawler source IPs, then change the narrow rule responsible for the failure.
Let one system own each schema entity
An SEO plugin may already output a WebSite, Organization, Person and Article graph. Inspect that output before inserting a second JSON-LD snippet in the theme. Adding another block with a different publisher name creates a conflict rather than a fix.
Set the actual site identity in the plugin that owns it. Give article authors real profile pages and visible bylines. Use supported filters or template hooks for missing fields instead of hard-coding a new Organization on every post. Plugin APIs differ, so use the installed plugin documentation for the exact filter.
<!-- Inspect the published page for a connected graph: -->
<!-- Article.author -> Person @id -->
<!-- Article.publisher -> Organization or Person @id -->
<!-- WebSite.publisher -> the same publisher @id -->
<!-- Headline, dates and identity must match visible page facts. -->Check the article without the visual editor
Page builders can publish a complete HTML document, or leave useful text dependent on client-side code. Test the actual public post. If the fetched body contains the answer, the fact that the editor uses JavaScript is irrelevant.
Use a descriptive post title, one main heading and sections that answer real reader questions. Keep canonical URLs on the intended permalink. Review the sitemap generated by WordPress or the SEO plugin, and avoid publishing competing indexes with different URL sets.
Add llms.txt only if you will maintain it
A small Markdown index can help a tool that chooses to consume your documentation. Serve it from /llms.txt as text, with a site name, a short description and links to canonical guides. WordPress rewrites must let this resource through before a catch-all page response.
Do not buy a plugin solely because a scorecard calls this file mandatory. This scanner gives it zero points. If you publish it, check the live body after caching and updates so the file does not quietly become an HTML 404 page with a 200 status.
# Actual site name
> Documentation for the people using this product.
## Guides
- [Setup](https://example.com/setup/): Configure the product.
- [Support](https://example.com/support/): Contact the maintainer.Recheck a homepage and a real post
Run the scanner on both routes after deployment. A homepage can pass while the article template carries a different author graph, noindex tag or client-only body. Compare reports only under the same scoring version and with complete measurements.
Save the report locally if you need a before-and-after record. A new score confirms what this scan measured. For crawler activity, inspect verified logs; for citations, inspect actual engine answers and their source links.
Check your site's technical readiness
Scan a public URL for crawler rules, delivered HTML and page metadata. The report distinguishes measured issues from checks it could not complete.
Check my site →