Foundational Usability Testing
- Formative Usability Testing
Conducted early in development to identify and resolve potential use-related issues through user feedback and observation.
- Summative (Validation) Usability Testing
Conducted on final, market-ready devices to validate that intended users can operate the device safely and effectively, per IEC 62366-1 and FDA guidance.
- Heuristic Evaluation
Expert review based on usability principles (e.g., visibility of system status, error prevention).
- Task Analysis
Breaks down user workflows to identify high-risk or error-prone steps.
AI-Specific Usability Considerations
- Explainability and Interpretability Testing
Evaluates whether the AI system explains its decisions in a way that is understandable and trustworthy to users (e.g., visual heatmaps, risk scores, confidence intervals).
- Confidence Calibration Testing
Determines whether users appropriately trust or question AI outputs based on provided confidence levels or uncertainty measures.
- Alert Fatigue / Notification Management Testing
Validates that AI-generated alerts or recommendations are clear, actionable, and not overwhelming.
- Decision Support Acceptance Testing
Measures how likely users are to follow AI-generated recommendations and under what conditions they override or ignore them.
- Bias Perception Testing
Evaluates user perception of fairness and reliability of AI outputs across diverse patient groups.
Human-AI Interaction Testing
- User Override and Manual Control Testing
Ensures that users can easily override AI actions or predictions when needed, with proper fail-safes in place.
- Human-in-the-Loop Workflow Simulation
Tests how clinicians interact with the AI in real clinical environments or realistic scenarios (e.g., triaging patients, diagnostics).
- Cognitive Load Testing
Measures mental workload and ease-of-use when interpreting AI outputs during real-world tasks.
- Response Time and Feedback Loop Testing
Validates the responsiveness of the system to user inputs or corrections.
Interface and Accessibility Testing
- Graphical User Interface (GUI) Usability Testing
Tests design clarity, intuitive layout, navigation, and readability of AI dashboards or decision-support tools.
- Voice/Touch Interaction Testing
Especially important for wearable AI devices, robots, or smart assistants used in surgical or patient-care settings.
- Accessibility Testing
Ensures the interface accommodates users with visual, motor, or cognitive impairments (per WCAG, Section 508, or EN 301 549 standards).
Usability Risk Management Testing
- Use Error Analysis
Identifies and categorizes user errors and their root causes (especially critical for AI systems where incorrect interpretation may be subtle).
- Use-Related Hazard Identification (linked to ISO 14971)
Maps device use scenarios to potential hazards caused by interaction with AI elements (e.g., incorrect diagnosis suggestions).
- Residual Risk Acceptability Testing
Assesses user tolerance and system responses to residual use-related risks.
User-Centered Design Validation
- Persona-Based Testing
Involves multiple representative user types (e.g., nurse vs. physician, novice vs. expert) to validate usability across all target audiences.
- Scenario-Based Usability Testing
Simulates real clinical scenarios using the AI system, with observation and feedback.
- Think-Aloud Protocol
Users narrate their thought process while using the AI tool—helpful for identifying friction or confusion in real-time.
Regulatory Usability Documentation
- Usability Engineering File (UEF)
Complete documentation of the usability engineering process (per IEC 62366-1).
- Human Factors Summary Report (for FDA submissions)
Includes test plans, results, analysis, and justifications that support safety and effectiveness.
- AI-Specific Usability Risk Report
Documents unique usability risks related to algorithm transparency, bias, and explainability.