Skip to main content
WCAGrules
Quick navigation

Guides · Testing and tools

The Report Tested a Configuration, Not Your Product

Every theme, flag and regional variant is another rendering somebody has to open. The ones nobody opened are the ones the report cannot speak for.

Last reviewed August 31, 2026

An audit covers the build somebody pointed it at. For a product that ships in configurations, that build is one configuration out of many, and the honest report names which one. Everything else on this page follows from that single sentence, whether you sell the product or bought it.

The standard already knows this. WCAG's rules for writing a conformance claim carry a note for web-based products that have no address until a customer installs them, and it says such a product may state that it would conform when installed. That is a conditional claim, deliberately. The thing you shipped and the thing your customer is running are two different objects, and only one of them has a URL somebody can test.

So the useful question is not whether the product is accessible. It is which layer of the stack a given finding belongs to, and who can change it. That is what the matrix below is for, and it is the artifact worth building before anybody quotes you for anything.

What a Conformance Claim Can Actually Cover

Conformance is defined only for web pages. A claim, though, may cover one page, a series of pages, or multiple related web pages, and that is written into the standard rather than being an interpretation of it. The number of pages is not the constraint people think it is.

The real constraint is evidence. W3C's own evaluation methodology says a claim covering many pages is available only where every page inside it was evaluated, or produced by a process that guarantees the result. Since most evaluations test a sample, most evaluations do not produce a conformance claim at all. They produce findings. That distinction survives every reframing, and a sample cannot answer for pages nobody opened.

One more boundary decides how far a claim reaches. A set of web pages needs both a shared purpose and the same author, group or organization, and W3C's own worked example of a separate set is a checkout that drops the site template. If your product's billing area is built by a different team on a different shell, that is not pedantry, it is a different set, and five of WCAG's rules are judged across sets rather than pages.

The Base Product and Implementation Matrix

This is the artifact. Nine layers, and for each one the two questions that decide a scope, which are who can change it and what a test of the base product can honestly say about it. Fill it in before you write a statement of work, because every row that says varies by customer is a row a single audit cannot close.

LayerWho changes itWhat a base-product test can sayWhat it cannot say
Application shell and chromeYou, the vendorThis is the strongest evidence in the whole stack, because every tenant gets the same markupNothing, if a customer can inject their own header or footer
Core componentsYou, the vendorComponent behaviour in the states the test reachedWhether an untested combination of components still works
Theme tokens (colour, type, spacing)The customer, usually through a settings screenThat the default palette measured as it did on the dayAnything about a palette the customer typed in afterwards
Layout options and density modesThe customerThe layouts that were openedThe ones that were not, including compact and expanded variants
Feature flagsYou, per tenant, often per releaseThe exposure that was live during the test windowAny branch that was switched off at the time
Regional and locale variantsBoth, depending on the fieldThe locales that were sampledRight-to-left rendering or a translated label nobody read
Customer-authored contentThe customer's staffThat the editor produced conforming output in the sampled casesWhether the next thousand pages of content do the same
Customer-installed integrationsThe customerNothing at all, until the integration is named and in scopeWhether the integration interferes with the rest of the page
Customer CSS and script overridesThe customerNothing, and this is the row that most often breaks a vendor claimWhether an override removed a focus style or an accessible name
Illustrative scoping matrix for a configurable product. The evidence column is what a base-product evaluation can support, not a promise about any real product.

Read the last three rows together and the commercial answer appears on its own. A vendor can carry the top of the stack for every customer at once, which is exactly why fixing something in the shared component library is the best-value accessibility work in software. Nobody can carry the bottom of it on somebody else's behalf.

Themes Are Configurations, and Untested Ones Stay Untested

Start with the part that is settled. WCAG's second conformance requirement says a full page includes each variation the page automatically presents for various screen sizes, and that each of those variations needs to conform on its own for the page to conform. A responsive product is therefore several tests, not one, and that is normative text rather than good practice.

Colour themes are not named in that sentence, so nobody should claim they are. What is true is the evidence point, and it is enough. A report that does not say which theme it tested cannot answer for the others, and dark mode is not a filter over light mode. It is a different set of computed colours, which means a different set of contrast results, and teams find that out when somebody measures.

Forced colors mode is the one that behaves least like the others, and it is worth understanding rather than guessing at. In that mode the browser enforces the reader's own palette over the colours you chose, for a specific list of properties, and it may paint a backplate behind text to keep it readable. The published list covers the obvious ones, color, background-color, border-color, outline-color, fill and stroke among them.

The two things forced colors will not repaint

box-shadow and background-image are not on that list. So a focus ring drawn as a box shadow keeps whatever colour you gave it while everything around it changes, and an icon painted as a background image keeps its own colours against a background that no longer matches. Both are invisible in a normal review and both are found in about a minute if somebody turns the mode on. The forced-color-adjust property can opt an element out with the value none, and the spec's own advice is to use that only where you are adjusting colours for the reader's needs yourself.

Test stateHow to reach itWhat tends to break
Default themeOpen the product as shippedThe baseline everything else is compared against
Customer-selected themeApply the palette the customer actually usesText and interface contrast, because brand colours were chosen for a logo
Dark mode by user preferenceprefers-color-scheme: dark at the OS levelElevation, borders and disabled states that were never redrawn
High contrast or increased contrast preferenceprefers-contrast at the OS levelSubtle dividers and placeholder text disappearing
Forced colors modeWindows contrast themes, or the browser's emulationFocus rings drawn with shadows, icons drawn as background images, and charts
Each responsive layoutResize to every breakpoint the product shipsRequired by conformance requirement 2, and the one people do remember
Illustrative theme test states. Each row is a separate pass, and each one produces its own findings.

Flags, Experiments and Regional Variants

A feature flag is a configuration that changes without a release, which makes it the hardest thing on this page to write a report about. The finding you recorded on Tuesday describes a build that may not exist on Thursday, and nobody involved did anything wrong.

There is a workable answer and it costs almost nothing. Record the flag state alongside every finding, the same way you would record the browser. A report that says observed with new-checkout on, beta-nav off, in the EU region is a report somebody can reproduce. A report that says observed on the checkout page is one your engineers will spend a morning failing to reproduce, and the argument that follows is about the report rather than the barrier.

Then keep two statuses apart, because they are not the same and reports routinely blur them. An observed barrier is something a person hit. An unevaluated branch is a variant nobody opened. Writing the second as though it were clean is how a report ends up saying more than the work behind it, and writing it as a failure is how a report loses its credibility. Both go in the document, in different columns.

VariantWhat decides who sees itIn this evaluationStatus in the report
new-checkout flagTenant setting, on for 12 of 40 tenantsTested with the flag onFindings recorded, flag state noted on each
beta-nav flagPercentage rolloutNot enabled during the windowNot evaluated, named as such
Pricing page experimentA/B split at the CDNVariant A onlyVariant B not evaluated
EU data-residency buildRegionSampledFindings recorded
Right-to-left localesLanguage selectionNot sampledNot evaluated, and flagged as a gap worth closing
Illustrative exposure inventory. One row per branch that can change what a reader receives.

If you want a defensible sampling story behind that inventory, WCAG-EM is the method to borrow from. It builds a structured sample from the common views and the essential functionality, then adds a random set sized at a tenth of the structured one, on top rather than carved out of it, and checks whether the random pages turned up anything the structured set missed. If they did, the structured set was not representative and you go back and widen it. That loop is the closest thing anyone has to a defence against a sample that flattered the product.

Reading a Vendor's Report Against Your Own Implementation

This is the buyer's half of the same problem, and it has a surprisingly good source. US federal buying guidance splits what it asks for by whether the item is standard off-the-shelf software or something built to order, and then carries a third case for off-the-shelf products that will be configured or modified to meet the contract.

For that third case the guidance suggests asking for the product's conformance report and a demonstration, in which the supplier shows how they will configure the solution to support the buyer's accessibility requirements. That is a procurement practice rather than a rule that binds you, and it is a good instinct to borrow, because it separates the thing the report covers from the thing you are about to switch on.

The same guidance asks, for ordinary off-the-shelf items, for a supplemental report carrying five things, one of which is information on how to configure and install the item to support accessibility. Read that requirement backwards and it tells you something useful. A product that needs configuring before it is accessible is a product whose report describes a configuration, and the configuration is a thing you have to be told about rather than discover.

  1. Read the report's scope statement before its rows. Every conformance report says somewhere what it covered, and that sentence controls everything below it. Our guide to what a VPAT cannot tell you covers the four things no edition of the template is structurally able to say.
  2. List what your tenant changed. Theme, flags, locales, integrations, custom fields, custom CSS. Each one is a difference between the tested product and yours.
  3. Ask which parts of your list the report covered. Not whether the product is accessible, which the supplier will answer honestly and unhelpfully, but whether the evaluation opened the configuration you run.
  4. Ask for the configuration guidance in writing. If there are settings that make the product more accessible, you want them named, because a default that fails and a setting that fixes it is a fifteen-minute job somebody has to know about.
  5. Keep the two piles apart in your own reporting. Findings your team can fix, and findings only the vendor can. Mixing them produces a backlog nobody can act on.

Writing the Limit Sentence Nobody Wants to Write

Whichever side of the deal you are on, the report needs one sentence that says what it does not cover. Vendors resist it because it reads like an admission. It is the opposite, because a report with no stated scope is a report a buyer cannot rely on, and buyers have learned to notice.

The shape that works names the build, the configuration, the environment and the date, and then says plainly what was outside. Evaluated on the 4.12 release, default theme, English locale, with new-checkout enabled and beta-nav disabled, in Chrome and Safari on macOS with VoiceOver and in Chrome on Windows with NVDA. Customer themes, right-to-left locales and third-party integrations were not evaluated. Four lines, and it turns a document people argue about into one they can use.

One honest limit

We audit and we never do the repair work, so nothing on this page is us offering to change your product. What we can do is evaluate a named configuration and write the findings so your team, or your vendor's, can act on them. If your product ships in many configurations, that is a conversation about scope before it is a quote, so start at pricing or send us the configuration list through contact.

Common questions

Does our vendor's conformance report cover our configured instance?
Only as far as its own scope statement reaches. Read that first, then list what your tenant changed, and ask specifically whether the evaluation opened that configuration. Federal buying guidance treats configured off-the-shelf software as its own case and suggests asking for a demonstration alongside the report, which is a good instinct even outside procurement.
Do we have to test dark mode separately?
If you want a report that can answer for it, yes. Dark mode is a different set of computed colours, so it produces a different set of contrast results, and a report that does not say which theme it tested cannot speak for the others. What is settled in the standard is narrower, and it is that each responsive layout the page presents has to conform on its own.
What about forced colors mode on Windows?
Worth a pass of its own, because it fails differently. The browser repaints a named list of colour properties with the reader's palette, and box-shadow and background-image are not on that list. Focus rings drawn with shadows and icons drawn as background images are the two things that usually go missing.
How do we report a finding on a page behind a feature flag?
Record the flag state on the finding, the way you would record the browser and the operating system. Then keep observed barriers and unevaluated branches in separate columns, because a variant nobody opened is neither a pass nor a failure and should not be written as either.
Can we make one conformance claim for all our customers?
Not from a sample. A claim covering many pages is available only where every page in it was evaluated or produced by a process that guarantees the result. What WCAG does offer a product vendor is the conditional form for software with no address before installation, which is a statement that the product would conform when installed.

Sources

Keep reading

More on testing and tools

Reading about it is the cheap part.

Find out where your site actually stands. The free scan checks 10 pages in a real browser against all 90 supported automated rules, keeps its 27 best-practice checks separate from WCAG findings, and names the rule behind every finding. The full audit adds an expert review and a real blind screen-reader user. From $499, with the report in 5 business days on Rapid and 10 on Standard, and the clock starting at cleared payment.

Go somewhere useful

Find tools, resources and your workspace.

29 destinations