Scoring methodology

Principles

1. Deterministic, or it does not ship. No language model participates in scoring. Tools that score AI visibility by sampling engine answers cannot be reproduced — ask twice, get two numbers — which means a difference between two runs carries no information, and comparing your page against someone else's is meaningless. Every check here reads the fetched bytes.
2. The arithmetic is public. Each check declares its weight and its pass condition. Nothing is a black box, and no factor is hidden because it is proprietary.
3. Subscores, not one number. A single figure averages "no crawler can reach this page" together with "the meta description is long". Categories are scored separately so a failure cannot be diluted by unrelated tidiness.
4. Fatal problems cap the grade. Without gates, a page no crawler can fetch still scores in the seventies because everything else is neat. See gate rules.
5. Skipped checks are declared. A check that does not apply is excluded from the denominator and named in the report, with the reason. Silently excluding checks lets two pages be scored out of different totals with nothing admitting it.

Categories & weights

Points do not sum across categories. A category's score is the share of its own applicable points that passed, and that percentage is then weighted by the figure in the Weight column. So Extractability having more checks than Crawler Access gives it no extra influence — its ceiling is 35% either way. This is worth stating because reading the Points column as a global total is an easy and reasonable mistake. Crawler access carries the most because it is a precondition: a page an engine cannot fetch has no other properties worth measuring.

The exact formula, so the score can actually be recomputed. For each category, categoryScore = Σ(weight of passed applicable checks) ÷ Σ(weight of all applicable checks) × 100. The overall score is the category scores averaged by the weights above, renormalised over the categories that had at least one applicable check: score = Σ(categoryScore × categoryWeight) ÷ Σ(categoryWeight) across those categories. Two consequences worth stating plainly: (1) because the denominator is only the applicable checks, a single check's real influence is categoryWeight × checkWeight ÷ Σ(applicable check weights in its category) — so on a page where it is the only applicable check in its category, it controls that whole category's weight. The Points column is its share when everything applies, not a fixed global contribution. (2) A category with no applicable checks at all (an evergreen page often has none for Freshness or Structured data) is excluded, not scored zero and not handed a free 100; its weight is redistributed in proportion across the categories that were measured, and the report shows it as N/A.

Gate rules

Some failures make the rest of the report academic. When any of these fails, the overall score is capped at 45 regardless of the other categories. The cap is a ceiling, never a floor — a page already scoring below it is not lifted.

• search_crawler_blocked — Search and live-browse crawlers are permitted by robots.txt
• meta_noindex — Page is indexable (no noindex directive)
• nosnippet — Content may be used in snippets and AI answers
• client_side_rendered — Primary content is present in the server HTML

The crawler gate is judged per engine, not per bot. Answer engines control their crawlers independently — blocking Perplexity says nothing about OpenAI, Google, Anthropic or Bing. So search_crawler_blocked caps the score only when the page is blocked to every answer engine at once, i.e. it is invisible to all of them. Being blocked to some but not all engines is a real loss but not a fatal one: it is reported as some_search_crawlers_blocked, a non-gate failure in Crawler access that names which engines are lost and which still reach the page.

client_side_rendered is a conservative gate. Most AI fetchers — OAI-SearchBot, PerplexityBot, the user fetchers — read the raw HTML and do not execute JavaScript, so content that only appears after a client render is invisible to them. Googlebot is the exception: it queues pages for rendering and can index script-generated content. The gate reflects the majority that cannot, so a page failing it may still be readable to Google while being unreadable to the rest.

Reports state when a score was capped and which check caused it, alongside the uncapped figure.

Grade bands

Bands exist because a raw number invites false precision. 87 and 84 are not meaningfully different pages; B and C are.

The 24 checks

Weight is in points. A check may claim more than one finding code when several distinct failures describe the same underlying property — no H1 and three H1s are different advice but one condition, and listing them separately would score the same property twice.

Crawler access 40% of the score

Extractability 35% of the score

Freshness 15% of the score

Structured data 10% of the score

Thresholds

Phrases like "substantive content" and "sized for retrieval chunking" are not reproducible descriptions — they are numbers. Publishing the checks without publishing these would make "anyone can recompute the score" a weaker claim than it sounds. Every value below is read from the same module the scorer uses, so the two cannot diverge.

Basis matters more than the value. experiment — Supported by published controlled experiment. mechanism — Follows from documented crawler or parser behaviour. judgement — Engineering judgement. Reasoned, not measured — argue with it. Most of these are judgement: reasoned, not calibrated against measured citation outcomes. Presenting them at the same evidential weight as a documented crawler behaviour would misrepresent how much is known.

Reproducible is not the same as valid

This methodology solves measurement stability, not measurement validity. A ruler that returns the same reading every time can still be measuring the wrong thing. The weights below are expert judgement about what matters, not coefficients fitted to observed citation behaviour — no ablation study, benchmark or cross-engine correlation has been run, and until one has, the honest position is that this is a well-specified opinion rather than a calibrated model.

The determinism claim should also be read precisely. It holds given the same HTTP response, the same audit date, and the same parser version. A page can vary by country, user-agent, CDN edge, A/B assignment or WAF state, and freshness checks move with the calendar. What is guaranteed is that identical input yields identical output — not that a site returns identical input twice.

Applicability

13 of the 24 checks are conditional. A missing FAQPage is not a failure on a page with no FAQ, and counting it as a silent pass would inflate the score as surely as counting it as a failure would deflate it. Conditional checks are removed from the denominator and listed in the report with the reason they did not run.

Crawler taxonomy

Two properties decide what a robots.txt rule actually buys you, and an earlier version of this document conflated them — telling sites that disallowing ChatGPT-User would remove them from ChatGPT's answers, when the vendor documents that agent as neither obeying robots.txt nor determining search eligibility. Both properties are now tracked per agent, sourced from vendor documentation.

A site that blocks GPTBot but allows OAI-SearchBot is making a deliberate, coherent choice: no training, still answerable. A site that blocks OAI-SearchBot has removed itself from ChatGPT's answers, and usually did not intend to.

Most user-triggered fetchers ignore robots.txt — but not all, so this is tracked per agent, not per role. OpenAI states of ChatGPT-User that "because these actions are initiated by a user, robots.txt rules may not apply", and that it "is not used to determine whether content may appear in Search"; Perplexity says Perplexity-User "generally ignores robots.txt rules". A Disallow for either is reported as an ineffective directive, not a block. Claude-User is the exception: Anthropic states its bots "respect 'do not crawl' signals by honoring industry standard directives in robots.txt", and that disabling Claude-User "may reduce your site's visibility for user-directed web search" — so there, unlike its OpenAI and Perplexity counterparts, a rule both takes effect and costs visibility.

Google-Extended is not training-only. Google describes it as limiting "AI training and grounding in some of Google's other systems". Blocking it does not affect Google Search, AI Overviews or AI Mode, but it can affect grounding in Gemini Apps and Vertex AI — so the cost is real, just not where most people assume.

robots.txt is evaluated per agent using Google's group-selection rules: an exact user-agent group replaces the * group rather than merging with it, path patterns support * and $, and the longest matching rule wins with Allow taking an equal-length tie.

This is the set of agents currently tracked, not the whole ecosystem. The score reasons about the 15 agents in the table above, covering the answer engines with the widest reach — OpenAI, Google, Anthropic, Perplexity and Microsoft. Others exist and are not yet modelled (for example Meta's user-triggered Meta-ExternalFetcher, and Amazon's Amzn-SearchBot and Amzn-User). A page can therefore be reachable or blocked to an engine this tool does not yet list; the crawler-access result should be read as "for the engines below", not "for every AI engine in existence".

What is not measured

• Whether any engine actually cites the page. That is a different measurement — observed visibility — and it requires querying the engines. A single query is not a stable score because the answer moves between runs, but controlled repeated sampling (fixed prompt, model, locale and sample size, reported with a confidence interval) genuinely can measure it. It is simply not what this tool does: this measures technical readiness deterministically, and mixing a sampled metric into a reproducible one would forfeit the reproducibility that is the whole point. The two are complementary, not rivals.
• Backlinks, domain authority, traffic, rankings. Not fetched, not inferred.
• Whether the content is accurate, useful or competitive. No judgement of quality is made or implied.
• JavaScript-rendered content, unless the rendering verdict says otherwise. The audit reads what a crawler that does not execute scripts would read.
• Anything behind authentication. Only what a public crawler could fetch.

Crawler access probes, when enabled, are labelled by confidence rather than folded into the score as fact. Probes originate from this server, not from a vendor's verified IP range, so a bot-verification challenge is the expected response to them and is reported separately from a genuine block.

Disagreements

If a weight looks wrong, a check looks unjustified, or a page scored in a way the rules above do not explain, that is worth hearing: [email protected]. A published methodology is only useful if it can be argued with.