AI-Assisted Clinical Audit: Navigating Opportunities and Governance

Introduction

Clinical audit, as defined by NICE, is a quality improvement process that seeks to improve patient care and outcomes through systematic review of care against explicit criteria and the implementation of change. This cyclic process ensures that practice aligns with best available evidence and national standards. While invaluable, the manual nature of many audit tasks can be a significant barrier to conducting comprehensive and timely audits, especially across large patient cohorts or multiple sites.

Artificial Intelligence (AI) refers to computer systems capable of performing tasks that typically require human intelligence, such as learning, problem-solving, perception, and decision-making. In the context of clinical audit, AI can assist by automating repetitive tasks, identifying anomalies, synthesising information from disparate sources, and ultimately, providing deeper insights faster than traditional methods.

Why this topic matters

NHS organisations face increasing demands to demonstrate quality, safety, and efficiency. Clinical audit plays a vital role in meeting these demands, but its effectiveness is often constrained by resource availability and the sheer volume of data involved. AI offers a pathway to:

• Increase Efficiency: Automate data extraction from electronic health records (EHRs), reduce manual charting and data entry, and speed up data analysis.
• Expand Scope: Enable audits to cover larger patient populations, more complex criteria, or previously unfeasible areas due to data volume.
• Enhance Insights: Identify patterns, trends, and deviations from standards that might be missed by human reviewers, offering a more nuanced understanding of care delivery.
• Improve Timeliness: Facilitate more frequent, near real-time audits, allowing for quicker identification of issues and implementation of corrective actions.
• Reduce Human Error: Minimise transcription errors and inconsistencies inherent in manual data handling.

Given the strategic drive towards digital transformation within the NHS and the ambitions of the NHS AI Lab, understanding and cautiously adopting AI in audit is becoming increasingly important for all clinical and quality improvement leads.

Practical explanation: How AI can assist in clinical audit

AI's utility in clinical audit primarily stems from its ability to process and interpret vast amounts of data more efficiently than humans. Here are several practical applications:

1. Automated Data Extraction

• Natural Language Processing (NLP): AI models can process unstructured clinical notes (e.g., discharge summaries, clinic letters, nursing observations) to extract specific pieces of information relevant to audit criteria, such as diagnoses, procedures, medication administrations, or test results. This can significantly reduce the need for manual chart review.
• Structured Data Querying: While not strictly AI, advanced analytics platforms often integrate AI capabilities. These can automatically query structured data fields (e.g., ICD-10 codes, SNOMED CT codes, lab results, prescribing data) from EHRs based on defined audit criteria.

2. Data Cleaning and Validation

• AI algorithms can identify inconsistencies, missing data points, or potential errors within datasets, flagging them for human review. This enhances the quality and reliability of audit data.

3. Pattern Recognition and Anomaly Detection

• Identifying Deviations: AI can detect when clinical practice deviates from established guidelines or expected norms. For instance, flagging patients who meet criteria for a specific intervention but did not receive it, or identifying unexpected lengths of stay for particular diagnoses.
• Predictive Analytics: While less common in audit (which is retrospective), AI could, in some contexts, flag patients at higher risk of adverse outcomes, allowing for proactive intervention and subsequent audit of the effectiveness of these interventions.

4. Report Generation and Visualisation

• AI-powered tools can automatically generate summary reports, create visualisations (e.g., dashboards, graphs, heatmaps) that highlight key audit findings, and even suggest areas for further investigation or improvement.

5. Prioritisation of Audits

• By analysing existing data on patient outcomes, complaints, or incidents, AI could help identify clinical areas most in need of audit, ensuring resources are directed where they can have the greatest impact.

Common pitfalls and crucial considerations

While the opportunities are significant, implementing AI in clinical audit is not without its challenges. Careful consideration and robust governance are essential.

1. Data Quality and Availability

• Garbage In, Garbage Out: AI models are only as good as the data they are trained on and process. Poor quality, incomplete, or biased data will lead to unreliable audit findings. Data cleaning and validation remain paramount.
• Access to Data: Secure and compliant access to large, diverse clinical datasets is fundamental. This often involves navigating complex information governance (IG) frameworks.

2. Algorithmic Bias and Fairness

• AI models can inadvertently replicate or amplify existing biases in the data they are trained on (e.g., demographic biases). This could lead to inequities in identifying care issues or, conversely, in missing problems for certain patient groups. Regular auditing of the AI's output for fairness is critical.

3. 'Black Box' Problem and Explainability

• Some sophisticated AI models (e.g., deep learning) can be difficult to interpret, making it challenging to understand why a particular finding was made. In clinical audit, understanding the reasoning is crucial for trust and for effective improvement. Prioritise 'explainable AI' where possible.

4. Information Governance and Data Security

• GDPR and Data Protection Act 2018: Handling patient data with AI solutions requires strict adherence to data protection principles, including purpose limitation, data minimisation, and robust security measures. DPIAs are often essential.
• Cyber Security: AI systems, like any digital system, are targets for cyber threats. Secure system design, regular audits, and robust access controls are non-negotiable.
• Cloud vs. On-Premise: Deciding where data is stored and processed with AI tools (e.g., NHS-approved cloud providers versus local servers) has significant IG and security implications.

5. Clinical Safety and Validation

• AI tools are medical devices (or components thereof) if they have a medical purpose. Ensure compliance with medical device regulations (e.g., UKCA marking) where applicable. Even without a medical device classification, rigorous validation of AI outputs against established human processes is essential before reliance.
• Human Oversight: AI should always assist, not replace, human clinical judgement and oversight in audit. A 'human-in-the-loop' approach is vital for reviewing flagged issues and interpreting complex findings.

6. Integration and Interoperability

• Integrating AI tools with existing EHRs and other clinical systems can be complex, requiring robust technical infrastructure and adherence to interoperability standards.

7. Cost and Resource Implications

• Developing or procuring AI solutions can be expensive. Consider total cost of ownership, including training, maintenance, and integration. Skilled staff (data scientists, AI specialists) are also required for implementation and oversight.

Step-by-step approach to implementing AI-assisted clinical audit

Embarking on AI-assisted audit requires a structured, iterative approach.

1. Define the Audit Scope and Objectives (Human-Led): Clearly articulate what you want to audit and why. Identify specific audit criteria. Start with a well-defined, manageable audit where AI might offer clear advantages.

2. Assess Data Availability and Quality:

   • Identify the data sources needed (EHRs, pathology systems, imaging systems, etc.).
   • Determine if the data is structured or unstructured, and its accessibility.
   • Conduct a preliminary assessment of data completeness and accuracy.

3. Conduct an Information Governance Impact Assessment (DPIA):

   • Thoroughly assess the data protection and privacy implications of using AI, ensuring compliance with UK GDPR and DPA 2018. Obtain necessary approvals from your DPO and Caldicott Guardian.

4. Proof of Concept (PoC) or Pilot Project:

   • Start small. Select a limited scope audit for a PoC. This allows you to evaluate the chosen AI tool's performance in a real-world setting without large-scale commitment.
   • Test the AI's ability to extract data and identify pre-defined criteria accurately.
   • Compare AI-derived results with traditional manual audit results (the 'gold standard') to validate accuracy. Establish clear metrics for success.

5. Iterate and Refine:

   • Based on PoC findings, iterate on the AI model (if internally developed) or configure the commercial tool. Address any discrepancies or biases identified.
   • Refine audit criteria or data processing methods as needed.

6. Develop Governance and Oversight Frameworks:

   • Establish clear responsibilities for human oversight of AI outputs.
   • Define processes for reviewing and challenging AI findings.
   • Plan for regular audits of the AI system itself to monitor its performance, fairness, and potential drift over time.
   • Ensure transparency, documenting how the AI works and its limitations.

7. Scale Up Gradually:

   • Once a pilot is successful and governance is robust, gradually expand the use of AI to more complex or broader audits.
   • Continuously monitor performance and gather user feedback.

8. Training and Education:

   • Provide adequate training for clinical staff, QI leads, and audit teams on how to use and interpret AI-assisted audit tools, and critically evaluate their outputs.

Example in clinical practice: Reducing Sepsis Mortality

Consider an NHS Trust aiming to reduce sepsis-related mortality by ensuring timely administration of antibiotics and appropriate fluid resuscitation, as per national guidelines (e.g., NICE guidance on Sepsis).

Traditional Audit: A manual audit would involve clinical audit staff retrospectively reviewing a sample of patient records coded with sepsis, looking for documentation of 'Sepsis Six' bundle completion within the mandated timeframe. This is time-consuming and often limited to a small sample size due to resource constraints.

AI-Assisted Audit:

1. Data Source Integration: The AI system is integrated with the Trust's EHR, lab results system, and medication administration records.
2. Automated Identification: An NLP model is trained to identify patients with suspected or confirmed sepsis from clinical notes, ED arrival documentation, and microbiology results.
3. Automated Data Extraction: For identified patients, the AI (via a combination of NLP and structured data queries) automatically extracts key data points:
   • Time of presentation to ED/hospital.
   • Time of 'Sepsis Six' elements initiated (e.g., blood cultures taken, broad-spectrum antibiotics given, lactate measured, fluid resuscitation started, urine output charted, oxygen administered).
   • Specific medication details (name, dose, route, time).
   • Physiological parameters (HR, BP, Sats, Temp, GCS) from vital signs charts.
4. Compliance Assessment: The AI compares the extracted timelines and interventions against the national 'Sepsis Six' guidelines and internal Trust protocols, automatically flagging records where compliance gaps are identified (e.g., antibiotics administered >1 hour after recognition).
5. Report Generation: The AI system generates a dashboard showing compliance rates, common points of delay, and specific patient cohorts where issues are more prevalent (e.g., elderly patients in specific wards). It links directly to the anonymised patient records for human review of flagged cases.
6. Human Review and Action: The clinical audit team and QI leads review the AI-flagged cases, investigate root causes, and develop targeted interventions (e.g., focused training for specific departments, protocol updates). Human clinicians remain responsible for interpreting the findings and making clinical governance decisions.

This approach allows for continuous, comprehensive monitoring of sepsis care, providing rapid feedback to clinical teams and significantly increasing the audit's impact on patient outcomes.

How Lazomis can help

Lazomis provides a structured framework for managing your quality improvement and audit projects, which is complementary to the adoption of AI-assisted tools. While Lazomis does not currently offer integrated AI analysis, it is designed to facilitate the 'human-in-the-loop' aspects crucial for effective AI integration:

• Project Management: Use Lazomis to define your AI-assisted audit projects, set clear objectives and criteria, and assign responsibilities for human oversight and review.
• Data Aggregation and Review: Our platform can serve as a repository for the outputs of your AI analysis, allowing teams to upload, review, and comment on AI-generated reports and flagged cases, ensuring appropriate clinical validation.
• Action Planning and Tracking: Document identified improvement actions, assign ownership, and track progress within Lazomis, ensuring that insights from AI-assisted audits translate into tangible improvements.
• Reporting and Dissemination: Consolidate your audit findings, including those derived with AI assistance, into comprehensive reports for CQC, governance committees, and wider dissemination. Easily demonstrate the impact of your improvement initiatives.

Using Lazomis alongside AI tools means you can leverage the efficiency of AI for data processing while maintaining robust human oversight, action planning, and continuous improvement cycles.

Key takeaways

• AI offers significant potential to enhance the efficiency, scope, and insights of clinical audit in the NHS.
• Key applications include automated data extraction, pattern recognition, and report generation.
• Rigorous information governance, data quality, and clinical safety processes are paramount for safe and ethical AI deployment.
• A 'human-in-the-loop' approach, where human clinicians oversee and validate AI outputs, is essential.
• Start with small, well-defined pilot projects (PoCs) to test and refine AI solutions before widespread implementation.
• This resource supports, but does not replace, clinical judgement. Local policy, formulary and specialist advice should be followed.