The Testing That Finds What Nothing Else Does

A page can satisfy all 55 criteria and still be exhausting to use. Every control named, every state announced, every contrast measured, and completing the task takes four minutes instead of thirty seconds because the order makes no sense to anybody working through it by ear.

That is not our complaint about the standard. It is the standard's own position. WCAG says in its introduction that content conforming at the highest level will still not be accessible to people with all types, degrees or combinations of disability, and it names cognitive, language and learning areas as where the gap is widest. It goes on to describe itself as testable with a combination of automated testing and human evaluation. Human evaluation was in the design brief.

Conformance testing cannot find effort, and neither can we by inspection. The only way to know is to watch somebody do it.

What It Finds That an Audit Does Not

• Effort. Two routes can both pass and take wildly different amounts of work. Nothing in the standard measures that, because no criterion is about how long a task takes.
• Sequence. Announcements that are individually correct and collectively baffling, usually because nobody ever listened to them in order.
• Recovery. What somebody does when they get lost, and whether the interface offers them anything at all at that moment.
• Assumptions. Wording that means one thing to the team who wrote it and something else to everybody else.
• Real configurations. People use assistive technology with their own settings, at their own speech rate, with years of habits attached. That is not the setup a tester recreates from a clean install.

Look at where the automated rule map is emptiest and you will find the same list. Focus order has no automated rule at all. Pointer gestures, pointer cancellation, dragging movements and target size have none between them. The guideline covering labels, instructions and error suggestions has nine criteria and a single proposed rule. Every one of those is about whether an interface can be operated rather than whether it is marked up, which is precisely what a session shows you and an inspection does not.

How Many Participants, and Which Ones

There is no number, and the absence is deliberate rather than an oversight. W3C's guidance says to get a few disabled people and then spends its words on who they should be, not how many. Where the count is concerned it points you at somebody else's book. So anybody quoting you a figure as a requirement is quoting a heuristic, and it is worth knowing which one.

The heuristic almost always meant is that five participants using a product in fairly similar ways will surface most of the serious problems in a flow. The same work is explicit about the catch, which is that once you are testing distinct groups you want three or four in each rather than five overall. That catch is the whole story here, because assistive technology users are not one group.

So the spread matters more than the number. One blind screen reader user does not represent disabled users, and they do not even represent screen reader users. That population alone splits across JAWS at 40.5% of primary use, NVDA at 37.7% and VoiceOver at 9.7%, with nine in ten also using a screen reader on a phone. A round that covers a screen reader, magnification, keyboard-only, voice input and somebody with a cognitive disability will tell you more with five people than ten sessions on one configuration ever will.

There is a second axis, and it is the one W3C names directly. How practised your participants are has to match who your users are. An internal tool for accountants wants advanced assistive technology users, because that is who will be driving it every day and their expectations are set. A public service somebody uses once, to apply for a benefit or book an appointment, wants novices, because a practised user routes around a problem so fluently that you will never see it happen. Those are W3C's own two examples, and they produce opposite recruitment briefs from the same budget.

Two orderings follow from that. Run an expert review first, so the obvious barriers are already fixed and the session is spent on what only a session can find rather than on a missing label anybody could have caught. And run small evaluations through the build rather than one formal study at the end, which is W3C's position as well as ours, because a finding that arrives after the launch date is a finding nobody can act on.

Recruiting, and Paying

Pay participants at the same rate you would pay anybody else for their time, and say the rate up front. Disabled people get asked for unpaid help with accessibility work constantly, and asking somebody to spend an hour improving your product for free is not research, it is an extraction. That is our position rather than anybody's published guidance, and we would rather state it than dress it up as a standard.

1. 1Recruit through disability organisations, user research panels that pay, and communities rather than through your own customers only, since your customers are the people who already got past the barriers.
2. 2Say what the session involves, how long it will take, and what technology they will need, before they agree.
3. 3Ask what they need to take part, and then provide it. Interpreters, extra time, breaks, a different platform.
4. 4Let people use their own device and their own settings. A borrowed laptop with default settings tests nothing real.

What to Ask Them to Do

Give tasks, not features. “Buy a pair of shoes in size nine” finds things. “Try the filter panel” finds nothing, because you have already told them where to look and what to think about.

Pick the tasks that matter commercially, and test the whole journey rather than the page. There is a conformance rule behind that as well as a commercial one. Where a page is part of a process, every page in the process has to conform for any of it to conform, and the standard's own worked example of a process is a shop checkout. So if the checkout is the thing, test the checkout, all the way to the confirmation, including the mistakes people make on the way.

One task worth setting deliberately. If your sign-in or your contact form has a CAPTCHA, put it in the session. Screen reader users report CAPTCHA as the single most problematic thing they meet on the web, ahead of everything else, and it is the sort of barrier that produces an abandoned purchase rather than a complaint.

Running the Session Without Steering It

• Ask them to think aloud, and then stop talking. The silences are where the findings are.
• Never say where something is. The moment you do, you have destroyed the thing you were measuring.
• When they get stuck, wait longer than is comfortable, then ask what they are trying to do rather than offering the answer.
• Do not defend the design. You are collecting evidence, not selling.
• Record what they do as well as what they say, because people are generous and will describe a struggle as fine.

What to Do With What You Learn

Some findings map onto a criterion, and those go into the same queue as the audit findings. Many will not map onto anything, and those are the valuable ones, because they are the problems no checklist was ever going to surface.

Write them as barriers rather than as opinions. “Three of five participants could not tell which delivery option was selected” is something a developer can act on, in a way that “users found the radio buttons confusing” is not.

Then write down who was in the room and what they were asked to do, at the top of whatever you produce. W3C's instruction is to record the scope and the evaluation parameters, and its own example is blunt about why. A study that ran only with blind participants has to say on its face that it did not evaluate conformance and does not speak for people with other disabilities. Otherwise the document outlives the context it was true in, and somebody six months later reads five sessions as a verdict on the whole product.

Then go back. Tell participants what changed because of what they told you. Almost nobody does this, and it is the difference between research and taking.

If You Cannot Run Your Own

Our audits include testing by people who use assistive technology daily rather than by sighted testers imitating them, which is the blind user testing service and part of every full audit. The distinction between that and a tool run is set out in automated versus manual testing.

It is not the same as usability research with your own customers, and it is not sold as a replacement for it. It answers a narrower question, which is whether the thing can be operated at all.