Accessibility Testing

In today’s world where digital services are widespread, accessibility of software and web-based applications to everyone has become a fundamental requirement. Accessibility is not merely a technical standard; it is also a concept directly linked to social participation, equality, and human rights. Within this context, accessibility testing is a systematic type of software testing conducted to evaluate the usability of software products by individuals with diverse abilities. The purpose of accessibility tests is to ensure that software is usable not only visually and auditorily but also in terms of cognitive, motor, and linguistic impairments.

Accessibility refers to the effective usability of information and communication technologies (ICT) by individuals regardless of their physical, sensory, cognitive, or neurological differences. This concept is not merely a matter of technical compliance but also a requirement for social participation and an integral part of user-centered design. The World Wide Web Consortium (W3C) defines accessibility as making websites, applications, and digital content perceivable, operable, understandable, and robust for individuals with disabilities.

Accessibility is not limited solely to physical impairments such as vision or hearing loss. It encompasses a broad spectrum of functional disability types:

• Visual Impairments (Vis): Color blindness, low vision, or blindness.
• Hearing Impairments (Aud): Varying degrees from mild hearing loss to total deafness.
• Physical/Motor Impairments (Phy): Reduced muscle control, coordination disorders, tremors, or paralysis.
• Speech Impairments (Spe): Difficulty speaking clearly.
• Cognitive Impairments:
• • Attention and Memory (A&M): Difficulty following long sequences of tasks or retaining short-term information.
  • Higher-Level Reasoning (HLL): Challenges with abstraction, planning, and decision-making.
  • Language and Numeracy (L&N): Difficulty comprehending and processing dense text or numerical content.

This classification is an enhanced functional version of the traditional five-category system developed by W3C (visual, auditory, physical, cognitive/learning/neurological, and speech). The structure proposed by Bai and colleagues^【1】 prevents testing from focusing exclusively on visual or physical impairments and enables a more holistic evaluation.

Accessibility encompasses not only access but also interaction and comprehension. Within this framework, accessibility must be evaluated across four fundamental dimensions:

• Perceivable: Information must be detectable by users (e.g., providing alternative text for visual content).
• Operable: Users must be able to control the interface through various means such as keyboard, voice commands, or touch.
• Understandable: Interfaces and content must be intuitive, simple, and user-friendly.
• Robust: The system must function reliably across different technological environments and assistive technologies (e.g., screen reader compatibility).

These principles are defined by the Web Content Accessibility Guidelines (WCAG) and apply not only to web platforms but also to mobile and desktop applications.

Accessibility is not a feature that can be added to software after development through testing; it is a design goal that must be integrated from the outset. Unfortunately, studies in the literature reveal that a significant proportion of software developers are either insufficiently familiar with accessibility guidelines or face difficulties applying them in practice.

Therefore, accessibility should be viewed as an integral component of usability, user experience (UX), and quality assurance (QA) processes. The increasing adoption of automated testing tools and the transformation of accessibility into a continuously testable attribute play a key role in this transformation.

Accessibility testing employs various methods and tools to evaluate the usability of digital systems by individuals with disabilities. These methods can be categorized based on scope, accuracy, cost, ease of application, and their impact on different disability types. Research in the literature demonstrates that accessibility testing should not be limited to a single method; the best results are achieved through the combined use of multiple complementary approaches.

Automated testing tools are programs that detect accessibility errors in software systems without human intervention. They typically identify visual and structural deficiencies such as HTML tagging issues, color contrast problems, or missing content descriptions. These tools are frequently preferred due to their low cost and speed. However, they cannot adequately assess semantic context or issues related to cognitive impairments.

Key tools include:

• Android Accessibility Scanner: Detects issues such as missing content descriptions, low contrast, and small touch targets in Android applications.
• Wave, aXe, HTML CodeSniffer: Automatically scan web-based accessibility issues and provide limited but effective checks for visual and physical impairments.

This method involves conducting accessibility checks based on international standards such as WCAG (Web Content Accessibility Guidelines). The tester manually evaluates each component of the system against a checklist.

Advantages:

• Ensures compliance with legal and regulatory requirements.
• Can be documented and audited in written form.

Limitations:

• Time-consuming.
• Prone to errors depending on the tester’s knowledge and experience.

In simulation-based testing, the user experience associated with specific disability types is mimicked. For example, conditions such as low vision, color blindness, or dyslexia are simulated to test how accessible the interface is under those conditions.

Examples include:

• Cambridge Simulation Glasses (color blindness visualization)
• WebAIM Low Vision Simulator, Funkify (digital simulation of various disability types)

These methods are effective in fostering empathy and helping designers visualize accessibility needs; however, they have limited capacity to generate quantitative data.

Assistive technologies are tools used by individuals with disabilities (e.g., screen readers, alternative keyboards). Testing through these technologies evaluates how well a software application works with them.

Common tools include:

• Screen Readers: NVDA (Windows), VoiceOver (iOS), TalkBack (Android)
• Switch Access: Alternative interaction methods designed for users with physical interaction difficulties.

These tests are valuable because they closely reflect real user scenarios, but their implementation requires time and expertise.

In these methods, accessibility experts evaluate software by creating specific scenarios or user profiles (personas). Techniques such as heuristic evaluation, cognitive walkthrough, and barrier walkthrough fall within this scope.

Characteristics include:

• The Barrier Walkthrough method systematically evaluates barriers users may encounter.
• Persona Testing tests software using scenarios representing the characteristic traits of users with different disability types.

While these methods enable a deep evaluation of accessibility, they require high expertise and significant time investment.

Accessibility testing methods are compared below according to the following disability types:

This table demonstrates that automated tools are insufficient for testing cognitive, numerical, and linguistic impairments, and this gap can only be addressed through expert-based or hybrid methods.

While accessibility testing aims to ensure that software systems are usable by individuals with disabilities, achieving this goal faces numerous technical, organizational, and methodological challenges. Findings reveal that accessibility is frequently neglected as a quality attribute in the software development lifecycle and that testing practices often contain significant gaps.

The majority of current testing tools and methods focus heavily on visual and physical impairments, while evaluations for cognitive, linguistic, auditory, and speech-based disabilities remain inadequate. Specifically, cognitive barriers such as attention and memory difficulties, deficits in higher-level reasoning, or challenges in language and numeracy comprehension are insufficiently represented in testing.

Although automated testing tools offer low cost and rapid results, they are limited by their inability to perceive semantic context, understand user intent, and rely solely on superficial rules.

For example:

• If multiple buttons are labeled “Remove,” an automated tool may detect the duplication but cannot determine which content each label refers to.
• Even if a content description exists, the tool cannot assess whether the description is meaningful or functionally relevant.

Expert-based accessibility testing, especially in cognitive disability assessments, is indispensable. However, these tests:

• Require intensive human resources,
• Can take hours per application,
• Require evaluators to have specialized training.

Additionally, organizing manual user tests, recruiting participants with diverse disability types, and systematically analyzing results represent significant cost factors.

In many software projects, accessibility testing:

• Is deferred until the end of the project,
• Is viewed as an unnecessary cost,
• Is not included in test scenarios.

Accessibility controls must be embedded from the beginning of the software development process, yet many developers are either unfamiliar with accessibility guidelines or lack sufficient support to implement them. Particularly in mobile applications, dynamic user interface generation, device diversity, and platform-specific interactions further complicate this process.

The use of simulations and assistive technologies is important for approximating real user experiences. However:

• Most of these tools simulate only visual or motor impairments.
• Effective simulation technologies for impairments such as speech, attention deficits, or decision-making difficulties are not widely available.
• Evaluations are often based on hypothetical scenarios rather than modeling real user experiences.

Moreover, assistive technologies do not always function in compliance with standards—for example, screen readers may fail to recognize certain custom components, reducing test reliability.

Equally important as detecting accessibility errors is reporting them in a way that developers can understand. However, many automated tools provide insufficient user-friendly error descriptions and improvement suggestions. The technical knowledge required to fix these errors often creates implementation challenges, especially for inexperienced teams.

The effectiveness of accessibility testing is directly related not only to which tools or methods are used but also to which disability types these tools can cover. While the W3C classification is commonly used to evaluate accessibility testing methods, an expanded and more detailed functional disability categorization system has been proposed.

This system draws attention not only to physical impairments such as vision or hearing loss but also to limitations based on cognitive, linguistic, and higher-level mental processes. This enables a deeper test scope and clearly defines which methods address which disabilities.

The following table presents the proposed functional disability types and brief definitions:

This classification differs from the classical W3C structure by breaking down cognitive impairments into three subcategories, enabling a more precise analysis in accessibility evaluations.

• Best coverage is achieved through expert evaluation methods. However, due to their high cost and expertise requirements, these methods are rarely used in projects.
• Automated testing tools provide effective checks for visual and physical impairments but are largely inadequate for complex disabilities such as attention, memory, higher-level cognitive processes, and speech.
• Checklist-based audits offer the most balanced approach, as they can be configured to align with a broad range of disabilities. However, they still depend on the auditor’s knowledge.
• Simulation tools are valuable for fostering empathy but have limited practical test validity and are mostly confined to visual impairments.
• Assistive technologies successfully test visual accessibility through tools such as screen readers. However, it must be remembered that these tools represent only specific user profiles and do not always ensure semantic adequacy.

Improving the effectiveness of accessibility testing requires more than selecting appropriate tools or methods; the timing and manner of their application are equally critical. Empirical studies and industry guidelines in the literature highlight the need for strategic practices to ensure sustainable and inclusive accessibility testing.

Below, best practices for enhancing accessibility testing performance are presented under systematic headings.

Accessibility is often relegated to the background in software development processes or tested only at the end of product development. However, the best practice is to integrate accessibility principles into all stages of the software development lifecycle—analysis, design, development, testing, and maintenance.

• During initial analysis: The needs of users with disabilities within the target audience must be identified.
• During design: Mockups and user flows must be developed based on accessibility standards (e.g., WCAG).
• During development: Developers must pay attention to criteria such as content descriptions, keyboard access, and contrast ratios.
• During testing: Both automated and manual accessibility tests must be applied, and findings must be integrated into the bug tracking system.

This approach transforms accessibility from a feature to be added later into a fundamental component of product quality.

Due to the multidimensional nature of accessibility, a single testing method is insufficient. Therefore, the best practice is to use complementary methods together. This strategy is also known as “method triangulation.”

Recommended hybrid structure:

• Automated tools (e.g., Android Accessibility Scanner, aXe): Provide fast and repeatable scans.
• Checklists (e.g., WCAG, WebAIM): Provide manual compliance checks against standards.
• User testing: Seesional issues are identified by observing real users interacting with the system.
• Expert evaluation: Complex or semantic-level problems are assessed by expert review.

This multi-layered structure ensures that accessibility testing is deep, not superficial, and sustainable.

Not every testing method covers all disability types equally. Therefore, when defining the testing strategy, tools and methods must be selected according to the target user profiles and functional disability types.

For example:

• For visual impairments: Screen reader simulations and contrast analysis tools are appropriate.
• For cognitive impairments: Expert-based persona testing and analyses of information density and task complexity are required.
• For physical impairments: Minimum touch target testing or keyboard navigation testing are prioritized.

Accessibility testing is not a one-time audit but a quality assurance step that must be integrated into continuous integration (CI) processes.

• Automated testing tools must be integrated into the CI/CD pipeline at regular intervals.
• Detected errors must be labeled as “critical error” or “improvement suggestion,” and resolution timelines must be tracked.

This approach makes accessibility sustainable and reduces the risk of neglect.

Research shows that the majority of developers and design teams are insufficiently familiar with accessibility guidelines and struggle to apply them in practice.

To overcome this challenge:

• Internal accessibility training programs must be organized,
• Practical content on WCAG, ARIA, and Android/iOS accessibility APIs must be provided,
• Accessibility criteria must be included in code reviews.

Education transforms accessibility from a “post-error fix” issue into a “preventive design” factor.

Accessibility testing must not only detect errors but also generate actionable information for developers. To this end:

• Errors must be clearly explained with examples and in technical terms.
• Recommended correction steps must be provided for each error.
• Prioritization must be applied (e.g., “critical,” “important,” “warning”).
• Feedback loops must be kept short; communication between design, development, and testing teams must be strong.