Where Automated Testing Stops and People Start

Run a scanner on a broken site and you will get a number. Run three different scanners and you will get three different numbers. That is not a flaw in any of them. It is what happens when each tool picks a different slice of a problem that is mostly not machine-decidable, and the standards body that publishes the rules says so itself, in as many words. Evaluation tools cannot determine accessibility. They can only assist in determining it, and human judgement is required.

This page sets out where the line falls, using W3C's own published rule set rather than anybody's marketing. It runs long, because the argument is worth more than the slogan and because the numbers people quote at each other are measuring different things.

What a Machine Is Genuinely Good At

Give a scanner the jobs it wins at, and it wins decisively.

• Measuring contrast, where the two colours are known. It is arithmetic, and a machine does it on every element in milliseconds. W3C's own automated rule for contrast attaches a caveat worth knowing, though. Passing the rule does not necessarily mean the criterion is met, because nobody has a clear method for judging legibility when only some of the pixels have enough contrast. Text over a photograph is exactly that case.
• Finding missing attributes. No alt attribute, no form label, no page language, no document title. Present or absent, no judgment needed.
• Checking heading sequence. An h2 followed by an h4 is a numeric fact. Tools report it, and WCAG sets no heading-order rule, so it lands as tidiness rather than a failure. Our free scan runs this as an additional best-practice check and labels it separately; it does not present heading order as a WCAG failure.
• Validating ARIA. Whether a role is real, whether the required attributes are present, whether a value is permitted.
• Coverage. A scanner checks every element on every page without getting bored. A human reviewing page four hundred is not the human who reviewed page one.

Those are real failures affecting real people, and finding them cheaply is worth doing. Start with a scan. Our free 10-page scan runs all 90 supported automated rules in a real browser and keeps its additional best-practice results separate from WCAG findings.

What a Machine Hands Back to a Person

Every item below is a WCAG requirement, and each one turns on meaning rather than on markup. A tool can read the markup perfectly and still have nothing to say about the question the rule is actually asking, which is why these come back to a person.

Notice the pattern. Every one of those is a pass in an automated report. A site can score green across the board and still be unusable, and that is not hypothetical. It describes most of the sites we audit that arrive with a clean scanner report attached to the brief.

Does That Line Move? Yes, and Not Where You would Hope

It is worth being exact here, because "a machine will never do this" is a claim with a poor track record. The line does move. PAC, the free checker most document teams run their PDFs through, added AI-assisted checks in its 2026 release that read a document's content and flag structure that does not match it, including a heading styled as a heading that was never tagged as one. That is a judgment call software used to hand straight back.

What is telling is where the same release stopped. It does not check whether alt text is correct, because that needs a language model it does not ship with. It does not check reading order, and it says a person still should. Its own documentation calls the AI results supporting information rather than a verdict on the standard. So the tool that moved this line furthest in the last year moved it around the edges of structure and left the meaning questions sitting exactly where they were.

That is the version of the argument worth making. Not that software can never judge meaning, which nobody can prove and which every vendor announcement is designed to make you doubt, but that the products shipping today draw the line in the same place the table above does, and they say so themselves in their own documentation.

W3C's Own Rule Set Says the Same Thing

The strongest evidence here is not ours and is not a study. It is the metadata W3C publishes on its own automated test rules, and it is worth going through slowly, because this is the part a sceptical buyer can check line by line.

Every one of those rules publishes what its result means for conformance. On 59 of the 87 live rules, a pass is recorded as "success criterion needs further testing". A failure is decisive and a pass is not, and that asymmetry is built into the format of every rule the standards body publishes.

Only four rules in the entire set map a pass to "success criterion is satisfied". Here is everything a machine can settle outright, on W3C's own reckoning. The page has a descriptive title. The page's language subtag matches its actual language. A part's language subtag matches its language. And no keyboard shortcut uses only printable characters. That is the whole list.

Then there is coverage, which is worse than the pass semantics. Of the 86 criteria in WCAG 2.2, 49 have no rule at all. Narrow it to the set an AA audit actually tests and 24 of the 55 have none. Narrow it again to rules W3C has approved rather than proposed, and 37 of the 55 have nothing.

Two of those gaps are worth naming, because they are where the money and the harm are.

The seizures guideline has zero coverage. Not thin coverage. None, across every criterion in it. So a scanner returns a clean report on the one guideline whose failure mode is a medical event, and the flashing-content rule is one of the four that break a whole page's conformance no matter whose script caused it.

Forms are the emptiest region of the map. The Input Assistance guideline holds nine criteria and there is exactly one rule across all nine, still at proposal stage. Labels or Instructions, the criterion sitting next to it, has nothing at all. Forms are where most sites make their money, and they are the part automation reaches least.

And where the rules are thickest, they measure presence rather than truth. Name, Role, Value carries fifteen rules, more than any other criterion in the standard, and of the ten that name it, seven test whether a name exists rather than whether it describes the control. A button whose accessible name is the literal word button passes all seven. The full account is on 4.1.2 Name, Role, Value.

None of that is a criticism of the rules. They are exact about what they test and honest about what they leave open, which is precisely why they can be quoted. What they do not support is reading a clean automated report as an answer to whether a site conforms.

The Number Behind the Number

The published figures do not agree with each other, and understanding why is more useful than picking a favourite.

The UK's Government Digital Service seeded a page with 143 known failures in 2017 and ran ten tools at it. The best single tool found 41%, counting manual inspection prompts as finds, and all ten together reached 71%. Nobody runs ten scanners, and even that aggregate left 29% of a page built specifically to be caught. Deque's own documentation for axe-core claims 57% on average, from anonymised audit data across 13,000 pages. Our technique-by-technique count runs far lower, and that one is ours rather than W3C's. W3C's library holds 432 entries, 342 techniques plus 90 documented failures. We graded 356 of them, and of those 10 are fully machine-detectable, which is 2.8%. Another 228 are partly detectable and 118 need a person throughout.

Here is the reconciliation, because those numbers are not in conflict. They have different denominators. WebAIM's survey of a million home pages found that 96% of all detected errors fall into six categories, and all six are trivially automatable. Low contrast, missing alt text, missing form labels, empty links, empty buttons, missing page language. So if you count defect instances on real pages, automation looks excellent, because a handful of high-frequency rule types dominate the count. If you count criteria, it looks poor, because 24 of 55 have no rule at all.

Both numbers are honest. They answer different questions, and the question a buyer is actually asking is whether they meet Level AA. That is a per-criterion question, so the per-criterion number is the one that answers it.

A Federal Regulator Has Said This Out Loud

This is not only a technical argument any more. In a complaint filed against an accessibility overlay vendor, the US Federal Trade Commission stated that no automated testing tool alone can determine whether a website meets accessibility standards, and that manual human testing is required.

Read that for what it is. A complaint is an allegation rather than a judicial finding, and that matter ended by consent order with nobody admitting or denying anything. What it shows is a federal regulator putting the position on the record as settled fact in its own pleading, sourced to the standards body and to the people who build the best-known free scanner. If you have a vendor telling you their tool answers the compliance question, that is the sentence to put in front of them.

How to Split the Work

Spend money on judgment. Spend nothing on counting. That is the whole rule, and everything below follows from it.

1. 1Automate the mechanical checks and run them in CI. Contrast, missing attributes, ARIA validity, and heading order if you want it, knowing WCAG does not require that one. Catch these before they ship, on every build, at no ongoing cost.
2. 2Review the judgment calls by hand. Alt text truth, link purpose, error message usefulness, focus order sense, heading accuracy. This is where an expert earns their fee, and it is where most of the standard actually lives.
3. 3Put a real screen reader user on your key journeys. Checkout, booking, sign-up, contact. Not a developer imitating one. Somebody who uses the technology every day and will show you where the flow collapses rather than where the markup is untidy.

That last one has a conformance rule underneath it, which is why we test journeys rather than pages. Where a page is one of a series making up a process, every page in that process has to conform for any of them to count. W3C's worked example is a shop checkout. A scanner looks at pages one at a time and has no concept of a journey, so it cannot answer that question even in principle.

Those three passes are the structure our audit runs on, in that order, for exactly these reasons.