ACR Evaluation Tips
When evaluating vendor-submitted Accessibility Conformance Reports (ACRs), procurement teams have a critical opportunity to assess the potential state of a product’s accessibility and guide further evaluation. An ACR is not a guarantee of accessibility conformance, but a well-prepared ACR provides valuable insights into the vendor’s efforts and the product’s current accessibility.
This information is based on years of experience working as a Contracting Officer’s Representative (COR) within the Federal Acquisition Service. I was a member of and led numerous internal and assisted acquisitions for external agency partners during my time at the General Services Administration. It was my aim to help both the buyers and suppliers understand their roles and responsibilities regarding accessibility as well as improve real world accessibility and outcomes for individuals using the products and services being purchased.
Here’s a breakdown of what buyers should scrutinize in vendor-supplied ACRs to make informed decisions:
- ACR/VPAT version and date - The version of the Voluntary Product Accessibility Template (VPAT) used and the report date are fundamental indicators of an ACR’s relevance and accuracy.
- Correct version: Ensure the ACR uses the most current and appropriate VPAT version. Using outdated versions can indicate a lack of up-to-date accessibility practices.
- Recent report date: Prioritize ACRs with a report date within six months of the bid submission or market research date. An older report (e.g., greater than 12 months) may not reflect the current state of the product, as digital accessibility is an ongoing effort.
- Described and appropriate testing method - A credible ACR will clearly outline the testing methodologies employed. Look beyond simple automated scans.
- Comprehensive testing: The ideal testing method should include a combination of automated testing, manual testing, and critically, the use of assistive technology and/or usability testing with people with disabilities. This holistic approach aligns with understanding true usability, as automated tools only catch a fraction of issues.
- Avoid over-reliance on automation: An ACR that lists only automated tools or scans as the testing method is a red flag. While useful, automated scans miss the majority of accessibility issues, particularly those related to human interaction and assistive technology compatibility.
- Quality of support tables - The support tables are where the vendor details their product’s conformance against specific accessibility criteria. This section requires thorough examination.
- Completeness: Verify that support tables are completed for all applicable Web Content Accessibility Guidelines (WCAG) 2.x Levels A and AA criteria, as well as all applicable Section 508 or other legal criteria for your region. Note: the “x” in 2.x is meant to encourage use of the latest published WCAG version rather than pinned to a version which may be outdated over time.
- Detailed explanations: For any instances where the product “supports with exceptions,” “does not support,” or “partially supports” a criterion, expect detailed explanations of the issues.
- Workarounds and roadmap: Exceptional ACRs will include information about workarounds or alternatives to ensure accessibility, along with an indication of a timeline or roadmap for correcting known issues. This demonstrates a proactive approach to remediation and a commitment to ongoing accessibility.
- Consistency and specificity: Be wary of inconsistent or partially missing information in these explanations, or vague statements of “full support” without further details.
- Alignment with ongoing accessibility commitments - Beyond the initial ACR, consider how the vendor demonstrates a commitment to ongoing accessibility throughout the contract lifecycle. This shifts accessibility from a one-time check to a continuous quality standard.
- Accessibility audit & remediation plan: Look for evidence that the vendor commits to a full accessibility audit against WCAG 2.x AA success criteria as part of onboarding, with a clear plan for compliance based on the audit.
- Ongoing accessibility standards: Ensure the vendor adheres to Section 508 and WCAG 2.x AA standards for new features and functionality introduced in the product/system. Ideally, this means 0 errors reported via automated scans and manual testing for new additions.
- Integrated testing in development cycles: The Quality Assurance Surveillance Plan (QASP) should detail how accessibility testing, using a combined approach of automated and manual methods (e.g., Accessibility Insights, DHS Trusted Tester), is integrated into every sprint. This indicates a proactive, rather than reactive, approach to accessibility.
By carefully reviewing these aspects of vendor-supplied ACRs and considering their broader commitment to accessibility, buyers can significantly reduce conformance risk and procure digital products and support from teams that have a genuine interest in developing products that are usable and inclusive for all.
If you would like an ACR Evaluation scoring template to use in your procurement, drop me a note using my contact form or via one of the social media links in the footer of the page.
