Automated vs. Manual Accessibility Tests

Accessibility is mandatory -- both technically and user-oriented. But how do you ensure that mobile websites are truly accessible? This is where the combination of automated and manual tests comes in.

Key Points:

• Automated tests: Fast, efficient, check technical errors (e.g. missing alt texts, colour contrasts).
• Manual tests: User-centred, check usability and context (e.g. screen reader tests, keyboard navigation).
• Combination: Both approaches complement each other ideally to combine technical precision with a human perspective.

Quick Comparison

CriterionAutomated TestsManual TestsSpeedVery highTime-consumingUser experienceLimitedVery goodTechnical errorsVery goodLimitedCostsInexpensive for repeatsMore expensive for individual reviews

Conclusion: Automated tests cover technical aspects, manual tests evaluate user-friendliness. Together, they provide a comprehensive solution for accessible websites.

Manual Testing Procedures for Websites

Automated Testing Tools

After discussing the fundamental testing methods, we now turn to automated approaches.

How Do Automated Tools Work?

Automated accessibility tests systematically analyse a website's HTML, CSS and JavaScript against established criteria. These tools examine the website structure and check, among other things:

• Technical base elements
• Semantic structure
• Colour contrasts
• Media properties

Advantages of Automated Tests

Automated tests bring many benefits to development teams:

AdvantageDescriptionEffectTime-savingFast analysisShorter test cyclesConsistencyStandardised evaluation criteriaConsistent resultsScalableParallel checks possibleEfficient analysesCost-effectiveLess manual effortLower testing costs

Limitations of Automated Tests

Despite their strengths, automated tools also have weaknesses:

• Lack of context understanding: Alt texts cannot be checked for meaningfulness.
• Limited user perspective: The real usability by users is not evaluated.
• Problems with complex interactions: Dynamic content or multi-layered user interactions may be inadequately tested.
• False positives: Some reported problems are not actual barriers.

Popular Automated Testing Tools

• Google Lighthouse: A free tool integrated into Chrome that provides detailed accessibility reports.
• axe DevTools: A solution from Deque Systems that integrates seamlessly into CI/CD pipelines and offers in-depth analysis.
• WAVE (Web Accessibility Evaluation Tool): A browser-based tool from WebAIM that visually presents test results and is easy to understand.
• SiteImprove Accessibility Checker: A premium tool with comprehensive analysis functions and clear action recommendations, ideal for large websites.

Automated tools are an important foundation for accessibility tests. However, they should be supplemented by manual tests to obtain a complete picture. More on this in the next section.

Manual Testing Methods

Manual tests are an important complement to automated approaches, as they can directly evaluate the user experience.

Fundamentals of Manual Tests

Manual accessibility tests rely on a structured approach carried out by trained testers. The most important methods include:

• Keyboard navigation: Ensuring all functions are usable without a mouse.
• Screen reader tests: Using tools such as NVDA or JAWS for Windows or VoiceOver for macOS to check accessibility.
• Zoom tests: Checking whether content is still readable and usable at 200% magnification.
• Colour contrasts: Direct assessment of colour contrasts in important interface areas.

Advantages of Manual Tests

Manual tests offer clear advantages through the human perspective:

AdvantageDescriptionEffectContext understandingTests consider the usage contextBetter assessment of the operating logicUser perspectiveRealistic scenarios that tools often overlookPractical problem identificationFlexibilityAdaptation to specific requirements possibleIndividual test strategiesQualitative analysisIn-depth evaluation of user-friendlinessConcrete improvement suggestions

Challenges of Manual Tests

Despite their strengths, manual tests come with some challenges:

• Time expenditure: A thorough review of a medium-sized website can take 15-20 hours.
• Costs: Qualified testers incur higher personnel costs.
• Subjectivity: Testers can give different assessments.
• Documentation effort: Results must be documented in detail, which is time-intensive.

Required Testing Competencies

Certain skills are indispensable for successful manual tests:

1. Technical understanding

Knowledge of WCAG 2.1 and HTML semantics is fundamental.

2. Experience with assistive technologies

Confident use of at least one screen reader and other assistive tools.

3. Analytical skills

Systematic problem identification and precise documentation of results.

4. Communication skills

Test reports and improvement suggestions must be clearly and understandably formulated so that developers and project managers can implement them.

Manual tests complement automated procedures and thus create a more comprehensive testing strategy.

Direct Comparison of Test Methods

Here we look at the strengths and weaknesses of both test methods side by side.

Accuracy of Problem Detection

Automated tests are excellent at finding technical errors, while manual tests are better suited to uncovering context-related problems. They allow the assessment of user-friendliness and semantic relationships. These differences also have implications for resource efficiency, which we will look at later.

Time and Cost Expenditure

Automated and manual tests differ significantly in terms of time and costs. Automated tests require a larger initial investment but offer fast execution and repeatability. Manual tests, on the other hand, cost less to set up but require more time per test run.

AspectAutomated TestsManual TestsInitial investmentHigh, mainly through specialised softwareLow, based on individual cyclesTime expenditureLow -- fast automatic executionHigh -- time-intensive manual workRepeatabilityVery high -- tests are easily repeatableRequires fresh effort each timeMaintenance effortRegular but efficient in operationVariable, depending on test scope

Integration into the Development Workflow

Automated tests can be easily integrated into CI/CD processes, making them ideal for continuous development cycles. Manual tests, on the other hand, are better suited to providing additional insights at important milestones. Both methods play an important role, depending on needs and objectives.

Method Comparison Table

CriterionAutomated TestsManual TestsDetection of technical errorsVery goodLimitedEvaluation of user experienceLimited insightVery goodScalabilityHighLowCost efficiencyAdvantageous for frequent testsCost-effective for one-off testsPrecisionHigh for technical criteriaDependent on tester experienceContext understandingLimitedVery goodFlexibilityLimitedVery highDocumentation qualityStandardised resultsPractical, detailed reports

Combined Use of Both Test Methods

The combination of automated and manual tests offers a more comprehensive assessment of accessibility by connecting technical precision with the human perspective.

Advantages of Combined Tests

Through the combined approach, both technical and user-centred aspects of accessibility are covered. Automated tests handle the technical review, while manual tests evaluate user-friendliness. Together they enable:

• Regular technical reviews through automated tests
• Context-related analyses through manual tests
• Efficient resource utilisation through optimised methods

This approach creates a solid foundation for a structured and effective test process.

Guidelines for Combined Tests

A well-coordinated workflow combines the strengths of both test methods and ensures systematic review:

PhaseAutomated TestsManual TestsDevelopmentIntegration into the build processReview of new featuresPre-releaseFull technical analysisDetailed usability reviewMaintenanceRegular automatic checksManual review every three monthsUpdatesRegression testsFocused analysis of changes