Skip to content
siterank.info
Sign in Start free

WordPress

How to Audit a WordPress Site for AI Search Readiness

A step-by-step audit of the things that decide whether machines can reach, parse and understand a WordPress site — with the evidence tier for each check.

By ankitkumarvig@gmail.com Published September 9, 2026 Last reviewed September 9, 2026 Sources verified September 9, 2026 12 min read

Most "AI search readiness" advice is a list of new files to publish. The useful version is duller and more effective: check that machines can reach your pages, that each page states plainly what it is and who published it, and that nothing in the stack is quietly returning the wrong thing. Google is explicit that its AI features need no additional requirements, no special markup and no new AI text files beyond ordinary SEO — so the audit below is mostly ordinary technical SEO, run with a machine reader in mind, plus a short section of genuinely emerging practice kept clearly separate.

Work through it in order. Each step says what to check, what a pass looks like, and what tier the recommendation sits at.

1. robots.txt — and the WordPress twist

WordPress does not ship a robots.txt file. It generates one on request: do_robots() sets a text/plain; charset=utf-8 content type, fires the do_robotstxt action, and builds a default body of User-agent: *, a Disallow for the admin path, and an Allow for /wp-admin/admin-ajax.php, then passes the whole string plus the blog_public value through the robots_txt filter. Since WordPress 5.3 it no longer emits Disallow: / when search engine visibility is discouraged; that case is handled by robots meta instead.

Three consequences that catch people out:

  • A physical file wins. If anything has written a real robots.txt into the web root, the web server serves that and WordPress never runs. Your robots_txt filters, and your SEO plugin's robots.txt editor, are then silently inert.
  • The virtual file depends on rewrite rules and on the server. A misconfigured nginx or Apache rule can return 404 for /robots.txt while WordPress is perfectly healthy. This is common, invisible from wp-admin, and worth treating as critical: it means no crawler ever sees your rules.
  • Only one plugin should own it. Yoast, Rank Math, AIOSEO and SEOPress all offer robots.txt editing. Two editors filtering the same output produce a file nobody intended.

How to check: request https://yoursite/robots.txt from outside WordPress and read the raw response — status code, content type, body. Not the preview inside a plugin's settings screen, which shows what the plugin intends to emit, not what the server returns.

What a pass looks like: HTTP 200, text/plain, the rules you expect, one owner, and a Sitemap: line pointing at a sitemap that actually resolves.

Remember what the file does and does not do. Google states robots.txt is not a mechanism for keeping a page out of Search: a disallowed URL can still be indexed if other sites link to it, appearing without a description. Use noindex or authentication for genuine exclusion. Tier: ESTABLISHED STANDARD for the protocol, OFFICIAL PROVIDER GUIDANCE for the indexing caveat.

For AI crawlers specifically, decide the three jobs separately — training, search indexing, user-triggered fetching — rather than writing one blanket rule. robots.txt for AI crawlers works through the syntax and the trade-offs.

2. Sitemaps

WordPress has generated XML sitemaps in core since 5.5. The index lives at /wp-sitemap.xml, core registers three providers out of the box — posts, taxonomies and users — and paginates at 2,000 URLs per sitemap page by default (filterable via wp_sitemaps_max_urls). The whole feature can be switched off with the wp_sitemaps_enabled filter, which defaults to true for public sites; when disabled, the rewrite rules remain in place and return a 404 rather than leaving the URLs to resolve to something else.

The interoperability problem is the one to check for. Every major SEO plugin ships its own sitemap and disables core's, usually serving an index at a different path. So:

How to check: request /wp-sitemap.xml and whatever path your SEO plugin advertises. Determine which one returns real XML, and confirm your robots.txt Sitemap: line points at that one. Then spot-check the contents: are noindexed URLs excluded? Are the URLs the canonical versions?

What a pass looks like: exactly one live sitemap index, referenced from robots.txt, listing indexable canonical URLs only.

Google is clear that a sitemap helps discovery but does not guarantee crawling or indexing, and that small, well-linked sites may not need one at all. Treat it as a discovery aid, not a lever. Tier: ESTABLISHED STANDARD (the sitemaps protocol) plus OFFICIAL PROVIDER GUIDANCE (no guarantee).

3. Indexability

This is the cheapest high-severity check in the audit, because the failure is binary and total: a page that is noindexed cannot be retrieved by a search-grounded system, however good it is.

Check, in this order:

  1. Site-wide visibility. Settings → Reading. If "Discourage search engines" is on, WordPress emits noindex via the wp_robots filter across the site. It is the single most common cause of a site vanishing after launch.
  2. Per-page robots meta. Your SEO plugin sets these. Look for templates and archive types that were noindexed years ago and forgotten.
  3. X-Robots-Tag headers. A CDN, a security plugin or a server config can add noindex at the header level, where nobody thinks to look. Check the response headers, not just the HTML.
  4. Staging leftovers. Basic auth, IP allowlists and "coming soon" plugins block crawlers entirely and look fine in a browser you are logged into.

How to check: fetch pages as an anonymous client and read headers and meta together. A logged-in view in a browser is not the audit.

Tier: ESTABLISHED STANDARD.

4. Canonicals

Google treats rel="canonical" as a strong signal, not a directive, and picks its own canonical if the signals conflict; redirects are described as a stronger signal than the link annotation, and sitemap inclusion as a weak one.

On WordPress the specific hazards are structural rather than exotic:

  • Two canonical tags. Core emits one, an SEO plugin emits another, and now the page contradicts itself. Count them in the rendered HTML — this is an output check, not a settings check.
  • Paginated archives canonicalising every page to page 1, which effectively hides the deeper content.
  • Parameterised URLs from filters, search or tracking, each generating a distinct crawlable URL.
  • Attachment pages, which many sites still expose as thin duplicate content.
  • Mixed HTTP/HTTPS or www/non-www canonicals after a migration.

What a pass looks like: exactly one canonical per page, self-referential unless there is a deliberate reason, absolute, HTTPS, and matching the URL in the sitemap.

Tier: OFFICIAL PROVIDER GUIDANCE.

5. Titles and descriptions

Google generates title links automatically from the page's <title> element, on-page headings, and references to the page elsewhere, and advises a distinct, descriptive <title> on every page, avoiding vague labels and boilerplate that varies by one word. Snippets are likewise generated from page content, with the meta description used when it describes the page better than an extracted passage.

For a WordPress audit that means checking output, not fields:

  • Every indexable URL has a unique <title>. Duplicates cluster in archives, paginated pages and product variants.
  • No page relies on the theme's default pattern alone (Post Title – Site Name repeated across ten near-identical posts).
  • Meta descriptions are unique where they exist. Google explicitly notes identical descriptions across a site are unhelpful.
  • The title matches what the page is actually about, because an inaccurate title is worse than an absent one for a machine trying to decide whether the page answers a question.

Tier: OFFICIAL PROVIDER GUIDANCE.

6. Heading structure

Headings do double duty: they are the outline a person skims, and they are the boundaries a retrieval system uses to cut a page into passages. Modern retrieval works at passage granularity, so a page whose sections are cleanly delimited and self-contained gives a system something it can lift.

Check for:

  • Exactly one <h1> per page, carrying the page's actual subject. Many themes put the site name in an <h1> on every template.
  • No skipped levels — h2 followed by h4 — which breaks the outline for anything parsing structurally.
  • Headings that state the subject, not a teaser. "Refund window: 30 days" is retrievable; "The details" is not.
  • Widgets and sidebars injecting headings into the outline, which is a theme problem rather than a content one.

The structural correctness is ESTABLISHED STANDARD (HTML semantics). The claim that better passage structure improves the odds of retrieval by an AI system is EMERGING PRACTICE: plausible mechanism, widely adopted, not confirmed by any provider. Passage-level content structure goes deeper.

7. Entity clarity

An answer engine has to work out what your organisation is before it can name it. That resolution is easier when the site is consistent and explicit about its own identity, and harder when the site assumes the reader already knows.

Audit the basics:

  • One name, used consistently. Not "Acme", "Acme Inc." and "AcmeCloud" scattered across templates, footer, About page and schema.
  • The name and what you do, stated in prose on the home page and the About page, in a sentence a machine could quote. Not implied by a hero image.
  • Disambiguation where the name is shared with another organisation or an English word — say the sector, the location, the category.
  • The same organisation details everywhere: name, logo, address, contact.

Tier: EMERGING PRACTICE for the retrieval benefit; the underlying practice of stating clearly who you are is simply good writing. Entity clarity covers the layers, and structured organization information covers the markup side.

8. Author and About pages

Google's guidance on creating helpful content asks whether it is self-evident who authored the content, whether pages carry a byline where one would be expected, and whether bylines lead to further information about the author. It frames this as part of demonstrating experience, expertise, authoritativeness and trust, and says trust is the most important of the four.

On WordPress the gap is usually mechanical rather than editorial:

  • Author archives are enabled but empty — no bio, no credentials, no links.
  • Posts display "admin" or a first name as the byline.
  • The About page describes a mission but never states who runs the organisation.
  • Contact information is a form and nothing else, giving a machine no parseable detail.

What a pass looks like: a real byline linking to a real author page with a real biography; an About page that names the organisation and the people; a Contact page with parseable details. Note the hard rule: if you do not have a credential, you do not invent one. Fabricated authors, credentials and reviews are out of scope for any legitimate audit remedy.

Tier: OFFICIAL PROVIDER GUIDANCE for bylines and author information as a helpful-content signal.

9. Structured data — usually your SEO plugin's job

Yoast, Rank Math, AIOSEO and SEOPress all emit JSON-LD, and each builds its own graph. The audit question is therefore rarely "should we add schema?" and almost always "is the schema that is already being emitted correct, and is there only one of it?"

Check the rendered HTML for:

  • How many JSON-LD blocks are present, and which plugin emitted each. Two plugins emitting overlapping Organization or Article graphs is a defect, and adding a third block is not the fix.
  • Whether the markup matches the visible page. Markup that describes content a user cannot see is a guidelines violation, not an optimisation.
  • Whether types are appropriate. FAQPage on a page with no FAQ, or Review markup for reviews you wrote about yourself, is misrepresentation.
  • Whether required-in-practice fields are populated for the types you use.

Google's own framing is worth holding onto: valid structured data makes a page eligible for features, and Google does not guarantee that features consuming structured data will appear.

Tier: ESTABLISHED STANDARD (Schema.org vocabulary), OFFICIAL PROVIDER GUIDANCE (Google's requirements and the no-guarantee statement). The claim that schema causes AI citations is not supported by any provider documentation.

Internal linking is the part of this audit with the most direct mechanical effect, and it is the part WordPress sites most often get wrong at scale, because posts accumulate and nobody revisits them.

Look for:

  • Orphan pages with no incoming internal link. If nothing links to it, discovery depends entirely on the sitemap, and Google is clear the sitemap does not guarantee crawling.
  • Weakly linked pages — one or two incoming links, usually from an archive.
  • Descriptive anchor text, rather than "read more" and "click here", which give a machine nothing.
  • Broken and insecure internal links, which are pure loss.
  • Whether your most important pages are actually the best-linked ones. They usually are not.

Tier: ESTABLISHED STANDARD for crawl discovery. Internal linking for SEO, retrieval and AI covers the structural approach.

11. llms.txt — optional, emerging, and last for a reason

llms.txt is a proposal for a Markdown file at /llms.txt giving a curated summary of a site and links to its important pages. It has real adoption, including by AI labs for their own developer documentation.

What it is not: documented by any provider as an input to retrieval or answer generation. Google states directly that its AI features require no new machine-readable files or AI text files. Presence is not ingestion.

Tier: EMERGING PRACTICE. Under the project's evidence rules, that means it must not be scored as a deficiency and must not be phrased as a fix. If you publish one, publish it because a curated map of your site is cheap and might help an agent orient — not because a tool told you it was missing. What is llms.txt? sets out the format and the limits.

The audit in one table

# Check Highest tier of the recommendation Severity if it fails
1 robots.txt resolves, one owner, correct rules ESTABLISHED STANDARD Critical
2 Exactly one live sitemap, referenced from robots.txt ESTABLISHED STANDARD High
3 Indexability: no accidental noindex, no stray headers ESTABLISHED STANDARD Critical
4 One correct canonical per page OFFICIAL PROVIDER GUIDANCE High
5 Unique, descriptive titles and descriptions OFFICIAL PROVIDER GUIDANCE Medium
6 Clean heading outline, self-contained sections ESTABLISHED STANDARD / EMERGING PRACTICE Medium
7 Consistent, explicit entity identity EMERGING PRACTICE Medium
8 Real bylines, author pages, About and Contact OFFICIAL PROVIDER GUIDANCE Medium
9 One correct schema graph, matching the page OFFICIAL PROVIDER GUIDANCE Medium
10 No orphans, descriptive anchors, no broken links ESTABLISHED STANDARD Medium
11 llms.txt if you want one EMERGING PRACTICE Not a deficiency

What this audit cannot tell you

It cannot tell you whether an AI system will cite you. Every check above removes a blocker or improves a page's odds at a retrieval step nobody outside the provider can observe. The chain from a crawler fetch to a citation has four separate transitions, each of which can fail — crawled is not cited works through them.

The only way to find out whether an answer engine names or cites you is to ask it, repeatedly, under recorded conditions, and to report the result as a rate with a confidence interval rather than a rank. That is a separate discipline from the audit, and the two should not be conflated: fixing a canonical does not entitle you to claim a visibility improvement.

Key takeaways

  • WordPress generates robots.txt at request time; a physical file, a rewrite problem or a second plugin can silently break it. Check the raw HTTP response.
  • Core sitemaps live at /wp-sitemap.xml and are usually disabled by an SEO plugin. Confirm which one is actually served.
  • Indexability and canonicals are the highest-severity, lowest-effort checks. Verify in output and headers, not in settings screens.
  • Titles, headings, entity clarity, bylines and internal links are what make a page legible to a machine that has never heard of you.
  • Structured data almost always belongs to an existing plugin. Audit it; do not add a second graph.
  • llms.txt is emerging practice, not a requirement, and must never be scored as a failure.

Official sources & further reading