Avoiding Overclaiming in Quality Improvement: A Guide for Clinicians

Why this topic matters

Accurate and honest reporting is the cornerstone of good practice in any professional field, and quality improvement is no exception. In healthcare, misrepresenting QI results can have significant consequences. It can lead to:

• Misguided Policy and Practice: If project effects are overblown, others might adopt interventions that don't deliver the promised benefits in their own contexts, wasting time and resources.
• Erosion of Trust: Overclaiming erodes trust among colleagues, management, and even patients, undermining the collaborative spirit essential for QI.
• Hindered Learning: True learning from QI relies on understanding what worked, what didn't, and why. Overclaiming obscures this learning process.
• Ethical Implications: Presenting results inaccurately, even unintentionally, can have ethical ramifications, particularly if it influences patient care decisions or resource allocation.

For junior doctors and trainees involved in QI as part of their training or portfolio, understanding how to report accurately is vital for professional development and demonstrating competence in this crucial area.

Practical explanation

Overclaiming often stems from a genuine desire to showcase the positive impact of hard work or to secure further support for an initiative. However, it’s essential to maintain scientific rigour and intellectual honesty. Here's what overclaiming looks like in practice:

• Attributing all improvements solely to your intervention: Failing to consider other contemporaneous changes or trends that might have influenced outcomes.
• Generalising results too broadly: Assuming what worked in a specific ward or clinic will work identically across an entire department, hospital, or trust without further testing.
• Confusing association with causation: Observing an improvement after an intervention and concluding the intervention caused it, without accounting for other variables.
• Cherry-picking data: Presenting only the most favourable data points while omitting less impressive or contradictory findings.
• Making definitive statements from limited data: Drawing strong conclusions from small sample sizes or short follow-up periods.
• Ignoring context: Not acknowledging the specific local factors (e.g., staff engagement, existing resources, patient demographics) that might have contributed to success and may not be replicable elsewhere.

Distinguishing QI from Research

It's crucial to remember that most QI projects are distinct from formal research studies. QI aims for local improvement and often uses methods tailored for rapid cycles of change (e.g., Plan-Do-Study-Act cycles). Research aims for generalisable knowledge and employs more rigorous methodologies (e.g., randomised controlled trials) to establish causation and external validity. While both are valuable, their objectives and typical methodologies differ, which impacts how results should be reported.

Common pitfalls

Even with the best intentions, clinicians can fall into traps that lead to overclaiming.

• Lack of a robust run chart or control chart: Without appropriate statistical process control (SPC) charts, it's difficult to distinguish true signal from common cause variation, leading to misinterpretation of trends.
• Insufficient baseline data: Starting an intervention without adequate baseline data makes it impossible to accurately assess the degree of change.
• Confirmation bias: A tendency to interpret results in a way that confirms pre-existing beliefs or desired outcomes.
• Pressure to demonstrate impact: Organisational or personal pressure to show 'success' can subtly influence reporting.
• Misunderstanding statistical significance vs. clinical significance: An observed change might be statistically noteworthy but not clinically meaningful, or vice-versa.
• Ignoring confounding factors: Failing to account for other interventions, seasonal variations, policy changes, or staffing changes that occurred concurrently with your QI project.

A Framework for Accurate QI Reporting

To avoid overclaiming and ensure credible reporting, consider this structured approach:

1. Clearly Define Your Aim and Measures

• Aim Statement: Ensure your project's aim is SMART (Specific, Measurable, Achievable, Relevant, Time-bound) and focuses on the local context. E.g., "To reduce waiting times for specialist diabetes dietetic review in the outpatient department by 20% by [Date]" – not "to revolutionise diabetes care."
• Outcome, Process, and Balancing Measures: Use a balanced set of measures. Outcome measures tell you if you achieved your aim, process measures tell you if the intervention was implemented as planned, and balancing measures check for unintended consequences.

2. Utilise Appropriate Data Visualisation and Analysis

• Run Charts/Control Charts: These are essential for visualising data over time and distinguishing special cause variation (a true signal of change) from common cause variation (random fluctuations). Learn to interpret rules for identifying signals.
• Annotation: Mark your charts with the dates of interventions and other relevant events (e.g., staff training, equipment failure, policy changes) to help understand context.
• Trend Analysis: Look at trends over time, not just 'before and after' snapshots.

3. Acknowledge Limitations and Context

• Scope: Be explicit about the specific population, setting, and timeframe your project covered. Do not imply broader applicability than your data supports.
• Contemporaneous Factors: Always consider and mention other factors that might have influenced your results. Did a national guideline change? Was there a leadership change? Did staff numbers fluctuate?
• Causation vs. Association: Be careful with language. Instead of "Intervention X caused a reduction in Y," consider "Following the implementation of Intervention X, we observed a reduction in Y" or "Intervention X was associated with a reduction in Y," and then discuss the plausible mechanisms.
• Small Sample Sizes/Short Follow-up: If your data is limited, state this and present conclusions cautiously.

4. Use Measured Language

• Avoid Absolutes: Words like "always," "never," "proven," or "definitive" are rarely appropriate in QI reporting.
• Focus on 'Improvement' rather than 'Cure': QI is often about incremental gains.
• Qualify Statements: Use phrases like "Our data suggests...", "It appears that...", "This localised intervention may contribute to..." or "Further work is needed to confirm...".

5. Seek Peer Review

• Before presenting or publishing, have colleagues, particularly those familiar with QI methodology and the project area, review your results and interpretation. They can often spot unconscious biases or missed confounding factors.

Example in clinical practice

Consider a junior doctor leading a QI project on an acute medical ward aiming to reduce the time from 'admission decision' to 'ward bed'.

Overclaimed Statement: "Our amazing project completely eliminated delays in patient flow. We now have patients on a ward bed within 30 minutes of admission decision, proving our new pathway is a game-changer for the entire hospital."

Problematic aspects:

• "Completely eliminated delays" – unlikely; overstates impact.
• Specific 30-minute target stated universally – may not be consistently met or sustainable; implies a level of precision not always achieved.
• "Proving our new pathway is a game-changer for the entire hospital" – generalises findings from one ward to the whole hospital without evidence.
• Fails to consider external factors (e.g., reduced admissions during a specific period, a new consultant starting).
• No mention of balancing measures (e.g., did patients get discharged too quickly, leading to readmissions?).

Accurate and Responsible Statement: "Our QI project on Ward C aimed to improve our patient flow by reducing the time from 'admission decision' to 'ward bed'. Following the introduction of a revised discharge planning checklist and daily bed management huddle, our run chart analysis shows a reduction in median time, from 120 minutes to 65 minutes, maintained over a 3-month period. We observed this trend change after week 6 of the intervention. Concurrently, a new pathway for early discharge of suitable patients was introduced Trust-wide during this period, which may also have contributed to the observed improvement. While promising for Ward C, further testing is required to understand the applicability and impact of these interventions in other areas of the hospital, particularly regarding any potential impact on our balancing measure of 7-day readmission rates, which remained stable during the project period. This resource supports, but does not replace, clinical judgement. Local policy, formulary and specialist advice should be followed."

Why this is better:

• States the specific aim and measures.
• Mentions the data visualisation (run chart) and observed change.
• Acknowledges a confounding factor (Trust-wide early discharge pathway).
• Uses cautious language ("We observed," "may also have contributed," "While promising for Ward C," "further testing is required").
• Addresses generalisability issues explicitly.
• Mentions balancing measures, even if stable.

How Lazomis can help

Lazomis provides several tools to support accurate and transparent QI reporting:

• Lazomis QI Project Setup: Guides you through defining clear aims, measures, and baseline data collection, which are critical first steps to avoiding overclaiming.
• Lazomis Data Visualisation: Offers intuitive ways to generate run charts and control charts, helping you interpret data trends correctly and distinguish signal from noise. This reduces the risk of misinterpreting short-term fluctuations as significant improvements.
• Lazomis Reporting Templates: Provides structured templates that prompt you to include sections on limitations, contextual factors, and a balanced interpretation of results, encouraging a comprehensive and honest account of your project.
• Learning Resources: The Lazomis library includes further resources on statistical process control and QI methodology, deepening your understanding of robust data analysis and reporting.

By leveraging these tools, you can systematically plan, execute, analyse, and report your QI projects with greater confidence and accuracy, ensuring the integrity of your findings.

Key takeaways