Ensuring Human Oversight in AI-Supported Improvement: A Clinician's Guide

Why this topic matters

AI tools are designed to automate and augment, but they do not replace clinical judgement. In healthcare, decisions often involve complex variables, ethical considerations, and unique patient contexts that algorithms alone cannot fully grasp. Erroneous AI outputs, if unchecked, can lead to patient harm, misallocation of resources, or erosion of trust.

For NHS organisations, integrating AI without sufficient human review is a significant clinical safety and governance risk. Regulatory bodies and national guidelines, such as those from NHS England and the MHRA, increasingly emphasise the need for robust validation, monitoring, and human-in-the-loop strategies. QI leads, clinical directors, and digital transformation teams are tasked with harnessing AI's potential whilst safeguarding patient care, making human review an indispensable cornerstone of any AI strategy.

Practical explanation

Human review in AI-supported improvement refers to the systematic process where human experts, typically clinicians or other relevant professionals, scrutinise, validate, and interpret the outputs, recommendations, or actions generated by an AI system. It encompasses several stages:

• Upstream Review (Design & Development): Clinical experts are involved in defining the problem, selecting appropriate data, validating algorithms, and establishing performance metrics. This ensures the AI is built on sound clinical principles and aims to solve a relevant, well-defined problem.
• In-Process Review (Monitoring & Feedback): Once deployed, human users actively monitor the AI's performance, checking for unexpected outputs, biases, or drifts. This involves regular auditing of AI decisions against ground truth and providing feedback to refine the system.
• Downstream Review (Decision & Action): This is perhaps the most critical stage, where a human clinician ultimately makes the final decision based on, but not dictated by, the AI's insights. The AI provides information or suggestions, but the human retains accountability and the authority to override the AI's output, particularly in patient-facing contexts.

This 'human-in-the-loop' approach acknowledges AI's strengths in data processing and pattern recognition while leveraging human strengths in critical thinking, ethical reasoning, and nuanced understanding of complex, real-world scenarios.

Common pitfalls

Implementing AI with insufficient human review can lead to several challenges:

• Automation Bias: Over-reliance on AI outputs, leading clinicians to accept recommendations without critical appraisal, even when they may be incorrect or inappropriate for a specific patient. This is a well-documented cognitive bias.
• Loss of Clinical Skills: A reduction in clinicians' active engagement with complex decision-making processes, potentially weakening their diagnostic or analytical skills over time.
• Unintended Consequences: AI systems can exhibit biases present in their training data, or their outputs might have unforeseen negative impacts on patient pathways or equity if not carefully monitored by diverse human teams.
• Lack of Traceability and Accountability: Without clear human responsibility checkpoints, it can be challenging to determine accountability when an AI-assisted decision leads to an adverse event.
• Difficulty in Ethical Navigations: AI struggles with ethical dilemmas that require human judgement, empathy, and an understanding of individual patient values and preferences.

Step-by-step approach to integrate human review

Integrating effective human review into AI-supported improvement projects requires a structured approach:

1. Define the Scope and Risk: Clearly identify where AI is being used and the potential impact of its outputs. Higher-risk applications (e.g., direct patient care, resource allocation) demand more intensive human review.
2. Establish Clear Roles and Responsibilities: Document who is responsible for reviewing AI outputs, who can override them, and who is accountable for final decisions. This should align with existing clinical governance structures.
3. Design for Interoperability and Transparency: Ensure AI systems provide outputs in a clear, understandable format, ideally with accompanying confidence scores or explanations. Clinicians need to 'see' the reasoning, not just the answer.
4. Implement Training and Education: Provide comprehensive training for all users on how the AI system works, its limitations, how to interpret its outputs, and the importance of critical human oversight.
5. Develop Robust Review Protocols: Create standard operating procedures (SOPs) for human review, including frequency, type of review (e.g., random sampling, outlier checking, targeted review for high-risk cases), and documentation requirements.
6. Mechanisms for Feedback and Iteration: Establish clear channels for users to provide feedback on AI performance, report abnormalities, or suggest improvements. This feedback loop is crucial for the ongoing refinement and safety of the AI system.
7. Ongoing Monitoring and Audit: Regularly audit the AI's performance alongside human review effectiveness. This includes monitoring for automation bias, drift, and unexpected outcomes. Periodic re-validation of the AI model is essential.
8. Ensure Governance and Legal Compliance: Work closely with your organisation's governance, information governance, clinical safety, and legal teams to ensure all AI deployments and associated human review processes comply with national guidance (e.g., NHS AI Lab, MHRA) and local policies.

Example in clinical practice

Consider an AI tool designed to predict waiting list escalation risk in an outpatient service. The AI analyses demographic data, referral urgency, and previous appointment history to identify patients at highest risk of breaching waiting time targets or deteriorating while waiting.

• AI Output: The AI flags 20 patients for urgent review each week, recommending specific interventions like an earlier appointment or a pre-clinic assessment call.
• Human Review: A senior administrator or clinician reviews this list. They consider factors the AI might miss, such as a recent change in GP notes not yet integrated, a known social vulnerability, or a clinician's specific request for a later review. They might adjust the priority, consult with the referring clinician, or add notes for human follow-up. Crucially, they validate the AI's recommendation, not just accept it.
• Outcome: The human reviewer approves 15 of the AI's suggestions directly, modifies 3 (e.g., changing intervention type), and overrides 2 based on additional clinical context. Feedback on the two overrides is logged for AI model improvement. The ultimate decision on patient management rests with the human clinician.

This iterative process ensures that patients receive appropriate care, resources are optimised, and the AI tool becomes more precise over time, all while maintaining clinical accountability.

How Lazomis can help

Lazomis offers tools that can support the structured integration of human review into AI-supported improvement projects:

• QI Project Setup: Use Lazomis to define the scope of your AI project, establish clear governance structures, and document the human review protocols within your project plan.
• Data Collection & Analysis: Our platform can assist in capturing data related to AI outputs, human review decisions, and feedback, enabling you to audit the effectiveness of your human oversight processes.
• Action Planning & Tracking: Track actions stemming from AI outputs and human review decisions, ensuring follow-up and accountability. Monitor the impact of interventions chosen by human clinicians based on AI insights.
• Reporting & Dashboards: Create custom dashboards to visualise AI performance alongside human oversight metrics. Track instances of overridden AI recommendations, identify patterns, and monitor for potential automation bias or drift, providing essential insights for ongoing improvement.

Key takeaways