Safe and Effective Use of AI in Healthcare Audit and Quality Improvement

Why this topic matters

Clinical audit and quality improvement are fundamental pillars of good clinical practice, driving continuous learning and service enhancement within the NHS. Traditionally, these processes can be time-consuming, involving manual data extraction, collation, and analysis. AI offers the promise of automating or significantly assisting these tasks, allowing teams to focus more on intervention design and implementation rather than data mechanics.

The widespread adoption of AI tools, from large language models (LLMs) to advanced predictive analytics, necessitates a clear framework for their introduction into clinical and operational settings. Without careful consideration of data privacy, algorithmic bias, transparency, and accountability, the potential benefits of AI could be undermined by unintended harms. For NHS organisations, ensuring the safe, ethical, and effective deployment of AI in audit and QI is paramount to maintaining public trust and delivering high-quality care.

Practical explanation

AI encompasses a range of technologies designed to simulate human intelligence. In the context of audit and QI, AI tools can primarily assist with:

• Data extraction and normalisation: Automating the retrieval of relevant data from electronic health records (EHRs) or other systems, standardising formats for analysis.
• Pattern recognition: Identifying trends, outliers, or deviations from expected pathways that might indicate areas for improvement or non-compliance with standards.
• Predictive analytics: Forecasting potential future outcomes or risks based on historical data, allowing for proactive interventions.
• Automated reporting: Generating draft reports, summaries, or visualisations of audit findings or QI project progress.
• Literature review assistance: Sifting through vast amounts of research to identify relevant evidence for QI initiatives.

It is crucial to remember that AI tools are assistive. They are designed to augment human decision-making, not replace it. Clinical oversight, human review, and ethical consideration must remain at the core of any AI-driven process.

Types of AI in Audit and QI

1. Machine Learning (ML): Algorithms that learn from data without explicit programming. For example, an ML model could learn to identify patients at higher risk of readmission based on historical patient characteristics and outcomes, informing a QI project focused on discharge planning.
2. Natural Language Processing (NLP): Enables computers to understand, interpret, and generate human language. In audit, NLP could extract specific clinical information (e.g., presence of pressure ulcers, falls risk assessments) from free-text clinical notes, which would otherwise require time-consuming manual review.
3. Generative AI (e.g., Large Language Models - LLMs): Can create new content, such as generating summaries of complex audit reports, drafting initial project plans, or suggesting potential interventions based on identified problems. While powerful, these tools require rigorous validation of their outputs.

Common pitfalls

Integrating AI into audit and QI is not without its challenges. Awareness of these common pitfalls is key to successful and safe implementation:

• Data Quality and Bias: AI models are only as good as the data they are trained on. Poor quality data, missing information, or inherent biases within historical data (e.g., reflecting historical inequalities in care) can lead to biased outputs or incorrect conclusions. This can perpetuate or exacerbate health inequalities.
• Lack of Transparency (The 'Black Box' Problem): Some advanced AI models can be difficult to interpret, making it hard to understand how they arrived at a particular conclusion. This can hinder clinical trust and accountability, especially when AI outputs inform critical decisions.
• Over-reliance and Automation Bias: The tendency for humans to uncritically accept the output of automated systems. This can lead to overlooking errors or failing to apply clinical judgement when AI suggestions are presented as authoritative.
• Data Security and Privacy: AI systems often require access to large datasets, including sensitive patient information. Robust data governance, anonymisation/pseudonymisation, and adherence to GDPR and national NHS data security standards are paramount.
• Lack of Clinical Validation: AI tools must be rigorously validated in the specific NHS context in which they are deployed. A tool that performs well in one setting may not translate effectively to another due to differences in patient demographics, clinical pathways, or data recording practices.
• Ethical and Legal Gaps: The rapid pace of AI development can outstrip regulatory and ethical frameworks. Organizations must proactively consider the ethical implications of AI use, particularly regarding fairness, accountability, and patient autonomy.
• Scope Creep: Starting with overly ambitious AI projects without adequate infrastructure, expertise, or clear objectives can lead to project failure and wasted resources.

Step-by-step approach for safe AI deployment in Audit and QI

This framework outlines a structured approach to integrating AI into your audit and QI processes safely:

1. Define the Problem and AI Opportunity

• Identify a clear audit or QI objective: What specific problem are you trying to solve? Where are the bottlenecks or inefficiencies that AI could realistically address? (e.g., reducing manual chart review time for a specific audit, identifying patients for pre-emptive intervention for a QI project).
• Assess feasibility: Does your organisation have the necessary data infrastructure, data quality, and initial expertise to support an AI solution? Start small with well-defined problems.

2. Establish Governance and Ethical Oversight

• Form a multidisciplinary working group: Include clinicians, QI leads, data scientists/analysts, IT specialists, information governance leads, and patient representatives.
• Conduct an Impact Assessment: Use a framework like NHS England's AI Health and Care Award's 'AI Ethics and Assurance Toolkit' or similar local governance processes. Consider clinical safety, data privacy, bias risk, and ethical implications early.
• Secure necessary approvals: This includes information governance, clinical safety, and local organisational approvals. Ensure compliance with GDPR, Data Protection Act 2018, and NHS Digital's 'Data Security and Protection Toolkit' (DSPT).

3. Data Preparation and Management

• Ensure data quality: Clean, complete, and accurately coded data are essential. Address missing values and standardise data formats.
• Anonymisation/Pseudonymisation: Implement robust processes to protect patient identifiable data, especially if data is being shared or processed by external AI modules.
• Bias detection: Actively look for and mitigate potential biases in historical data that could lead to unfair or inequitable AI outputs.

4. AI Tool Selection and Development

• In-house vs. Vendor: Decide whether to develop AI solutions internally or procure them from a third-party vendor. If procuring, rigorously vet vendors for compliance, security, and proven track record.
• Transparency and Explainability: Prioritise AI models or vendors that offer transparent workings ('white box' models) or clear explanations for their outputs where possible.
• Prototyping and piloting: Begin with small-scale pilots. Define clear success metrics and safety parameters.

5. Rigorous Validation and Testing

• Clinical validation: Crucially, involve clinical experts to validate the AI's outputs against established clinical standards and human judgement. Does the AI make sense clinically? Is it accurate and reliable in your specific context?
• Real-world testing: Initially run AI tools in 'shadow mode' – processing data alongside human review without influencing real-world decisions directly, to compare performance.
• Performance monitoring: Continuously monitor the AI's performance for drift (where performance degrades over time due to changes in data or environment) and unintended consequences.

6. Implementation, Training, and Iteration

• User training: Provide comprehensive training to all users on how to interact with the AI tool, understand its outputs, recognise its limitations, and what to do if an error is suspected.
• Integrated workflows: Design AI integration into existing audit and QI workflows seamlessly, ensuring staff feel supported rather than replaced.
• Feedback loops: Establish clear mechanisms for users to provide feedback on the AI tool's performance and any issues encountered. This allows for continuous improvement and iteration.
• Continuous Governance: Maintain ongoing oversight and re-evaluation of ethical and safety considerations as the AI system evolves.

Example in clinical practice

A large NHS Trust aims to improve its management of sepsis, specifically reducing time to antibiotic administration. Historically, this has involved multiple clinicians manually reviewing emergency department notes for sepsis screening criteria and documenting times. This process is prone to human error and delays data for QI analysis.

AI Integration:

1. Problem Definition: Reduce manual data extraction time for sepsis audit and provide near real-time flags for potential sepsis cases based on screening criteria.
2. AI Solution: The Trust procures an NLP-driven AI tool, which integrates with its EHR. The tool is trained to identify key sepsis indicators (e.g., NEWS2 score, temperature abnormalities, suspected infection) from free-text and structured data within ED notes.
3. Governance: A multidisciplinary team, including emergency medicine consultants, microbiologists, QI leads, and IG, obtains local approvals. A clinical safety case is developed, detailing potential failure modes (e.g., false positives/negatives) and mitigation strategies (e.g., all AI flags require human clinical review).
4. Data Quality: The IG team ensures robust anonymisation for off-site training data, and the trust cleans its local structured data for training.
5. Validation: During a pilot, emergency clinicians review AI-generated summaries and flags. Initially, the AI identifies 85% of actual sepsis cases (true positives) and has a false positive rate of 10%. Clinicians validate these against their gold-standard assessments.
6. Implementation: The AI tool is deployed to assist ED clinicians. It generates a 'potential sepsis alert' within the EHR, but critically, it does not automatically start treatment. Clinicians are trained to use the alert as a prompt for urgent clinical assessment and decision-making. For QI, the AI automatically extracts relevant time stamps for antibiotic administration, streamlining the audit process and providing more immediate feedback on compliance with targets.
7. Outcomes: The trust observes a significant reduction in the average time taken to compile audit data for sepsis. More importantly, the AI alert system is linked to a measurable improvement in time to antibiotics for sepsis patients, improving patient outcomes. Continuous monitoring identifies a slight drift in performance after a change in EHR interface, prompting retraining of the AI model and a review of the alert logic.

This example highlights how AI acts as an assistant, improving efficiency and supporting clinical decision-making, while never replacing the essential human element of clinical judgement and oversight.

How Lazomis can help

Lazomis provides a structured platform that can significantly enhance your organisation's ability to manage and conduct audits and QI projects, whether or not AI is directly integrated into data collection. For teams looking to leverage AI, Lazomis offers:

• Structured Project Management: Organise your AI-driven audit or QI project from inception through to implementation and evaluation. Document your governance approach, risk assessments, and ethical considerations within the platform.
• Data Integration (planned): While Lazomis does not directly perform AI analytics, our platform is designed for secure, structured data input and clear visualisation. Future integrations will allow for more seamless data exchange with validated AI tools, ensuring that quality outputs from AI can be directly fed into your QI cycles.
• Compliance and Governance Tracking: Maintain a clear audit trail of decisions, approvals, and validation steps for your AI initiatives, ensuring adherence to local policies and national guidelines.
• Performance Monitoring Frameworks: Utilise Lazomis's dashboard and reporting features to track key performance indicators for your QI projects, including those influenced by AI interventions. Compare AI-assisted outcomes against baseline performance and monitor for 'AI drift' over time.
• Collaborative Workflows: Facilitate secure collaboration among your multidisciplinary AI working group, centralising documentation and communication around AI project development and deployment.

This resource supports, but does not replace, clinical judgement. Local policy, formulary and specialist advice should be followed.