Catching Regressions Before They Ship

We keep telling people that a scanner fully settles only 10 of the 356 WCAG techniques and failures we graded, which is 2.8% by our own count. That is a reason to be careful about what you claim from it, not a reason to skip it.

A machine will check those rules on every commit, forever, without getting tired. No human review process does that. So the point of putting checks in CI is not to prove conformance. It is to stop the same failures coming back after somebody has paid to have them fixed.

And the trend argues for a net rather than a one-off clean-up. WebAIM's annual survey of a million home pages found detected failures on 95.9% of them, averaging 56 per page, and that average rose 10% year on year. Sites drift back. A pipeline is the cheapest thing that notices.

What to Run, and Where It Fits in the Pipeline

The rules engine most tools share is axe-core, tuned so hard against false positives that its maker's own claim is zero of them, bugs notwithstanding. That property is what makes it safe to fail a build on.

Component-level checks are where most of the value is. They run in milliseconds and they catch the failure at the moment somebody introduces it, which is when it is cheapest to fix.

One thing they cannot do, and it is worth knowing before you rely on them. A component rendered in isolation has no page behind it, so the contrast rule has no background to measure against and will either go undecided or return a pass that means nothing. Contrast is the most prevalent detected failure in the field by a distance, so a suite that catches everything except contrast is missing the biggest single category. Cover it at the page level instead.

The Part Everyone Gets Wrong

Testing the default state and nothing else.

A scanner sees the page as it sits. Your closed menu, your unopened modal, your untriggered error state and your loaded-but-empty results list are all invisible to it. So a suite that only loads pages will pass a site whose every interactive component is broken.

Drive the state, then assert. Open the menu and check. Submit the invalid form and check. Open the modal and check.

The Bucket Your Assertion Is Probably Throwing Away

axe returns results in four buckets, not two. Violations, passes, inapplicable, and incomplete, which means the engine started a check and could not decide.

Almost every hand-written CI assertion looks at violations and nothing else. That is how the incomplete bucket disappears, and what lands in it is disproportionately what matters. Text over an image or a gradient, where the background cannot be resolved. Anything the engine can see but cannot judge. A pipeline that discards those is reporting confidence it does not have.

You do not have to fail the build on them, and you should print them. A count of undecided checks alongside the violation count keeps the report honest and points a human at the right places.

The Rules Worth Turning On Deliberately

Filtering to the WCAG tags is the sensible default, and it silently excludes a set of rules that are labelled best practice rather than conformance failures. Several of them are worth having anyway.

• Heading order. WCAG has no heading-sequence rule, so an h2 followed by an h4 is untidy rather than a failure. It is still a reliable smell, and a cheap one to keep clean.
• Empty headings, which are a real problem for anybody navigating by heading list even though no criterion names them.
• Label from title only, which catches the field whose only labelling is a placeholder or a title attribute. That is the case worth having most, because such a field genuinely satisfies the naming criterion and fails a different one that has no automated rule anywhere.
• Region, skip link and one-main-landmark, which between them describe whether a page has a usable structure to move around in.
• Positive tabindex, which is almost always a mistake and almost never intentional.

Add them as warnings first. They are not conformance failures and should not be reported to anybody as though they were.

Keeping It Honest

• Never let a green pipeline become a conformance claim. Put that sentence in the readme, in those words, because somebody will eventually paste the badge into a tender response.
• Fix, do not exclude. Every rule somebody disables to get a build green is a rule that stops protecting you.
• Record what is excluded, and why. An undocumented exclusion list quietly becomes the reason an audit finds things your pipeline never mentioned.
• Know whose rules you are enforcing. Of the 70 WCAG-tagged axe rules, 49 map to a W3C conformance-tested rule and 21 are the vendor's own. Those 21 find real problems and they are not conformance judgments, which is another reason the pipeline is not a claim.
• Re-run after fixes. The point is regressions, so verifying the fix closes the loop.

Why the Green Badge Means Less Than It Looks

There is a precise version of this, published by the standards body, and it belongs in that readme.

W3C maintains automated test rules for WCAG, and each one states what its result means. On 59 of the 87 live rules, a pass is formally recorded as "success criterion needs further testing". Only four rules in the whole set clear a criterion outright. So a pipeline full of passing rules means, by the standards body's own definition, that a large amount of further testing is needed.

Coverage tells the same story. Of the 86 criteria in WCAG 2.2, 49 have no automated rule at all, and 24 of the 55 at A and AA have none. W3C's own guidance on choosing tools says it plainly, that evaluation tools cannot determine accessibility and can only assist, and that human judgement is required.

What It Will Never Catch

The judgement calls, as always. Whether alt text is true, whether link text means anything out of context, whether focus order matches reading order, whether an error message helps, whether checkout can be completed by ear.

That last one is structurally out of reach rather than merely hard. Conformance for a process runs across every page in it, so a checkout only conforms if each of its steps does, and a per-component test suite never sees a process. It sees components.

Our automated versus manual guide works through why each of those resists automation. The split to aim for is simple enough to say in one line. Let the machine own the mechanical rules permanently, and spend human time on judgement.