Accessibility Testing by Machine, by Expert, and by Ear

What We Keep Finding

Run a free scanner and you get a number. Run three of them and you get three different numbers, because every tool draws its own slice of the standard and none of them draws the whole thing. That has actually been measured. A government team seeded one page with 143 known failures across 19 categories and ran 10 tools at it. The best single tool found 41% of them, counting the ones it merely flagged for a human to look at. All ten together found 71%.

We graded 356 of the 432 techniques and failures W3C publishes, and 10 can be settled by a machine outright. The other 346 need a person, either because a tool can only catch some of the cases or because it cannot see the thing at all. Our own free scan is the honest version of that limit, because it touches 20 of the 55 Level A and AA rules and leaves 35 of them untested.

Here is the mechanism, and it is worth understanding before somebody tells you the scan came back clean. Take a form field labeled by nothing but its placeholder. The rules for working out a control's name fall through to the placeholder, so the field has a name, so it passes the scanner's label check. What it fails is a different criterion about labels and instructions, and that one has no automated test rule at all. Layering the passes is what closes gaps like that. The machine finds the image with no alt attribute in about a second, and it is welcome to it. The expert reads the alt text and notices it describes the wrong product. Then the screen-reader user shows you what that actually costs, because they are the one who gives up and leaves.