A chat interface can be evaluated the way any other interface can. Launch it, reach it by keyboard, hear what it announces, close it, get focus back somewhere sensible. That work is ordinary and it finds real barriers.
What the chat produces is a different object. Every reply is content generated after the evaluation ended, from a prompt nobody wrote in advance, and an audit cannot certify content that did not exist when it ran. That is not a hedge, it is the same limit that applies to comments, reviews and anything else a page acquires later, and it is worth stating in the report rather than leaving for somebody to discover.
So the scope splits cleanly. The interface gets tested. The generated output gets tested in the states you can reach, with the states you could not reach named, and the shape of the output gets checked rather than its truth. Accessibility work has nothing to say about whether the answer was correct.
What Gets Tested in the Interface
The interface half is bigger than the bubble in the corner suggests, because a chat widget is a small application with a launcher, a panel, a growing transcript, a composer and a set of controls, each of which fails in its own way.
| Task | How to reach it | What tends to go wrong |
|---|---|---|
| Find and open the chat | Tab from the top of the page | A launcher built from a div with no role, or one that never appears in the tab order at all |
| Know the panel opened | Open it with the keyboard and listen | Focus stays on the page behind, so a screen reader user hears nothing change |
| Read what is already in the transcript | Arrow through the panel | History that is only reachable by scrolling with a pointer |
| Send a message | Type in the composer and submit | A composer with no label, or one where Enter both sends and inserts a newline depending on state |
| Hear a reply arrive | Wait, without moving focus | Nothing is announced, or every token is announced separately |
| Use a quick-reply button | Tab to the suggested answers | Buttons whose text means nothing out of context, and a set with no grouping |
| Attach a file | Activate the attach control | A drop zone that is the only route, covered on our file upload page |
| Reach a human | Ask for a handover | The wait state and the queue position are never announced |
| Close the chat | Escape, or the close control | Focus lands back on the launcher rather than where the reader started |
Why Announcements Are the Hard Part
Almost every difficult finding in a chat audit is about what gets said out loud and when. Status Messages is the criterion in play, and it is narrower than teams assume, because it is scoped by a defined term rather than by common sense.
Content counts as a status message only if it does two things. It has to provide information on the success or results of an action, the waiting state of an application, the progress of a process, or the existence of errors. And it must not be a change of context. So a connecting to an agent indicator is a status message. An error banner is a status message. A typing indicator is a waiting state, which is on the list.
The reply itself is a harder call, and the honest answer is that it looks more like a search result than a status message. W3C draws exactly that line for search, where the results are not a status message but the count of them is. The safe finding on a chat reply is about whether it can be found and read at all, not a claim that the criterion requires it to be announced.
One detail worth knowing before you file a bug
role="alert" carries an implicit live region. role="alertdialog" does not. They read like a pair and they behave nothing alike, so a widget that raises an error in an alertdialog and expects it to announce itself has a silent bug that no visual review will catch. If you are building rather than buying, our notifications and toasts guide covers which role does what.
Streaming makes this worse in a way the criteria were not written for. A reply that arrives token by token inside a polite live region can be announced dozens of times as it grows, and the result is unusable without failing any specific rule. Record it. Put it in the report as an observation with a reproduction, keep it out of the criterion column unless it maps on the facts, and let the product team decide. A report that inflates observations into failures loses the argument on the first one somebody checks.
A Published Example of How This Gets Written Up
The UK government's One Login service publishes an accessibility statement that lists its webchat separately from the rest of the service, with each problem mapped to a criterion. It is a useful model precisely because it is somebody's honest self-report rather than a marketing document, and because it separates two kinds of finding.
The mapped failures cover ground any chat audit recognises. Voice control software may not label the webchat content, filed under Name, Role, Value. External keyboard focus may not reach the question and answer section and can disappear, filed under both Keyboard and No Keyboard Trap. Some browsers move focus to the start of the chat when new content arrives, filed under Focus Order. Quick answer options are hard to tell apart with a screen reader, filed under Info and Relationships. Answer text in buttons is not descriptive out of context, filed under Headings and Labels. The field label is removed while somebody types their own reply, filed under Labels or Instructions.
Then it does the thing most reports skip. It lists three more problems under a heading saying they are not believed to be non-compliant but may be frustrating. You cannot change language mid-session. You cannot save the session. The chat does not confirm to a screen reader user that it has closed. Three real problems, honestly reported, correctly kept out of the failure column. That is the shape to copy.
Dates matter with anything like this. That statement was last reviewed in June 2024, its webchat was last tested in April 2024, and the audit behind it was carried out in January 2024 against WCAG 2.1. It is a snapshot of one service at one time, not a current description of anybody's product, and it should not be read as either.
The Generated Output, and What Can Honestly Be Said About It
The second subject is what the assistant sends back, and there are three useful things an evaluation can do with it.
- Test the shapes the output can take. A plain paragraph, a list, a table, a code block, a link, a form, a card with buttons, an image. Each of those is a rendering path in your widget, and each can be exercised with a prompt that reliably produces it. A table that arrives without header cells is a defect in the renderer, not in the model.
- Test the states around the output. Empty, loading, partial, complete, failed, rate-limited, refused. Those are states the interface owns, and they are where the announcements live.
- Record what could not be reached. If a handover to a human needed an agent who was not on shift, that state is unevaluated. Say so. An unreached state is not a pass.
What no evaluation can do is tell you the next answer will be accessible. Accessibility findings are about the interface and its rendering, and the model's behaviour is neither. Nor does any of this touch accuracy, safety or fairness, which are real concerns with their own methods and none of them is a WCAG audit.
The Widget Is on Your Page, Whoever Wrote It
Most chat is a vendor's script running on your domain, and the conformance rules are unsentimental about that. Conformance is claimed for full pages, and it cannot be achieved if part of a page is excluded, so there is no version of the argument where the widget is somebody else's problem and the page around it is clean.
For content that arrives later from sources you do not control, WCAG gives exactly two routes and no third one. Either the page is monitored and non-conforming content is repaired within two business days, in which case a determination of conformance can be made, or you publish a statement of partial conformance naming the uncontrolled parts clearly enough that a reader can identify them. The standard adds that no claim can be made at all if monitoring and correcting is not possible. Our third-party scripts guide works through what that means for widgets in general.
One honest limit
We audit and we never do the repair work, so we will not be changing your widget or your prompts. What an evaluation produces is the interface findings with reproductions, the output shapes that broke, and a named list of states nobody could reach. Where the widget is a vendor's, those findings are written so you can send them on. Scope it through contact, or see what the chat widget audit covers.