Choosing the Right Metrics for Healthcare Improvement

Why this topic matters

Effective healthcare improvement, whether through Quality Improvement (QI), clinical audit, or service evaluation, hinges on robust measurement. Without carefully chosen metrics, efforts can be misdirected, resources wasted, and the true impact of changes may remain unknown.

Poor metric selection can lead to several challenges:

• Misleading conclusions: Metrics that don't accurately reflect the desired outcome can lead to erroneous interpretations of performance.
• Demotivation: If metrics don't show the intended improvement, even when progress is being made, teams can become disheartened.
• Lack of demonstrable impact: Difficulty in proving the effectiveness of an intervention, making it hard to secure further investment or wider adoption.
• Gaming the system: If metrics are easily manipulated or don't align with clinical reality, staff may focus on 'hitting the target' rather than improving patient care.

Careful selection of metrics ensures that improvement work is evidence-informed, transparent, and aligned with patient needs and organisational goals.

Practical explanation

Metrics are measures used to quantify performance, processes, or outcomes. In healthcare, they can range from clinical indicators (e.g., readmission rates, infection rates) to operational measures (e.g., waiting times, bed occupancy) and patient reported experience measures (PREMs).

When choosing metrics, several key characteristics should be considered:

Validity

Is the metric measuring what it intends to measure? A valid metric accurately reflects the construct it's designed to assess. For example, if you want to measure patient safety, a metric like 'falls with harm' is more valid than 'total falls' if your intervention specifically targets preventing injury.

Reliability

Will the metric produce consistent results under consistent conditions? A reliable metric minimises random error. If different individuals collect the same data using the same metric, they should arrive at similar results. This requires clear definitions and standardised data collection methods.

Sensitivity and Specificity (where applicable)

In diagnostic contexts, sensitivity refers to the ability of a metric to correctly identify true positives, while specificity refers to its ability to correctly identify true negatives. While often applied to diagnostic tests, the concept can be extended to process metrics – how well does a process metric pick up actual deviations or successes?

Utility/Actionability

Is the metric useful for decision-making and driving improvement? Metrics should provide insights that can be acted upon. Complex, hard-to-interpret metrics have low utility. The metric should be relevant to the team undertaking the work and allow them to understand the impact of their changes.

Feasibility

Can the metric be collected practically and affordably? Data collection should not overly burden staff or require excessive resources. Consider existing data sources before creating new ones.

Alignment

Does the metric align with strategic objectives, national priorities (e.g., NHS Long Term Plan, GIRFT recommendations), and local needs? QI projects should contribute to wider organisational goals.

Common pitfalls

• Too many metrics: Overwhelm teams with an excessive number of measures makes it difficult to focus and interpret results. A 'vital few' are usually more effective than the 'trivial many'.
• Choosing metrics of convenience: Selecting metrics simply because the data is readily available, rather than because they are truly meaningful for the project.
• Lack of clear definitions: Ambiguous definitions lead to inconsistent data collection and unreliable results.
• Ignoring process measures: Focusing solely on outcome measures without also tracking the processes that lead to those outcomes makes it hard to understand why things are changing.
• Neglecting patient experience: Failing to include patient-reported outcome measures (PROMs) or patient-reported experience measures (PREMs) overlooks a critical perspective on care quality.
• Not involving the right people: Developing metrics in isolation without input from front-line staff, patients, or data experts can lead to impractical or irrelevant choices.
• Measuring an issue that is already resolved: Before starting, ensure the problem still exists and hasn't been addressed by previous work.

Step-by-step approach to metric selection

1. Define your aim statement: What exactly are you trying to improve, for whom, by when, and by how much? A clear, specific, measurable, achievable, relevant, and time-bound (SMART) aim statement is foundational. For example: "Reduce the average length of stay for elective hip replacement patients from 5 days to 3.5 days by November 2024, without increasing complications or readmissions."

2. Brainstorm potential measures: Based on your aim, what are all the different things you could measure? Consider:

   • Outcome measures: What is the ultimate effect you want to see? (e.g., readmission rates, patient satisfaction, mortality).
   • Process measures: What steps in the care pathway are you changing or want to monitor? (e.g., compliance with a new checklist, time to antibiotic administration, percentage of patients receiving pre-operative education).
   • Balancing measures: What are the unintended consequences (positive or negative) that might occur as a result of your change? (e.g., for reduced length of stay, balancing measures could be an increase in complications, patient complaints, or bed occupancy elsewhere).

3. Evaluate potential measures: For each brainstormed measure, ask:

   • Is it directly linked to the aim?
   • Is it valid and reliable?
   • Is it feasible to collect?
   • Will it provide actionable insights?
   • Is everyone involved clear on its definition?

4. Select a 'vital few': You typically need 1-2 outcome measures, 2-3 process measures, and 1-2 balancing measures. Resist the urge to measure everything. Focus on the measures that will provide the most insight into your improvement work. Prioritising these will help to maintain focus and prevent data fatigue.

5. Operationally define each metric: Write down precise definitions for each selected metric. This includes:

   • Numerator: What exactly are you counting?
   • Denominator: What is the total population or activity from which the numerator is drawn?
   • Inclusion/Exclusion criteria: Which cases are included or excluded?
   • Data source: Where will the data come from? (e.g., electronic patient record, theatre log, patient survey).
   • Frequency of collection: How often will data be collected? (e.g., daily, weekly, monthly).
   • Who is responsible for collection?

6. Pilot and refine: Before full implementation, test your data collection methods and definitions. Are there any ambiguities? Is the data clean? Adjust as necessary.

This resource supports, but does not replace, clinical judgement. Local policy, formulary and specialist advice should be followed.

Example in clinical practice

QI Project Aim: To reduce the average waiting time for urgent suspected cancer referrals (2-week wait) from first presentation to initial specialist assessment from 10 days to 7 days by 31st March 2025, for patients attending the X hospital trust, without increasing patient distress or misdiagnoses.

Brainstormed Measures:

• Patient waiting time (presentation to assessment)
• Number of cancelled appointments
• Percentage of appointments attended
• Patient experience survey scores for waiting times
• Number of complaints related to waiting times
• Diagnostic accuracy rates (misdiagnosis)
• Time to diagnosis
• Time from referral to first investigation

**Selected Metrics (Vital Few):

Outcome Measure:

• Average time (days) from urgent suspected cancer referral (2WW) to first specialist assessment.
  • Operational definition: Number of calendar days from the date the GP referral is received by the trust to the date of the patient's first face-to-face or virtual consultation with a specialist for the suspected cancer. All 2WW referrals for suspected cancer, excluding those where the patient defers appointment. Data source: PAS/EPR. Frequency: Monthly.

Process Measures:

• Percentage of 2WW referrals triaged within 24 hours of receipt.
  • Operational definition: (Number of 2WW referrals triaged within 24 hours / Total 2WW referrals received) * 100. Data source: Central referral system. Frequency: Weekly.
• Percentage of 2WW referrals offered an appointment within 3 working days.
  • Operational definition: (Number of 2WW referrals offered an appointment within 3 working days / Total 2WW referrals received) * 100. Data source: PAS/EPR. Frequency: Weekly.

Balancing Measure:

• Patient Reported Experience Measure (PREM) for anxiety levels experienced during the waiting period.
  • Operational definition: Average score from a validated 3-item anxiety scale (e.g., adapted GAD-2) completed by patients at their first specialist assessment. Data source: Patient survey. Frequency: Monthly, for a random sample of 50 patients.
• Number of formal complaints related to 2WW pathway.
  • Operational definition: Total number of formal complaints logged in the complaints management system where the primary reason is explicitly related to or impacts the 2WW cancer pathway. Data source: Complaints system. Frequency: Monthly.

How Lazomis can help

Lazomis provides a structured framework within its QI Project Setup tool to guide you through the process of defining your aim and selecting appropriate metrics. Our platform allows you to:

• Document clear metric definitions: Standardise your data collection with dedicated fields for numerator, denominator, data source, and frequency.
• Track and visualise your measures: Easily input and display your chosen outcome, process, and balancing measures on run charts or control charts within Lazomis Dashboards, helping you to interpret variation and understand the impact of your interventions.
• Collaborate with your team: Share your chosen metrics, data definitions, and progress with all relevant stakeholders, fostering transparency and shared understanding.
• Support reporting: Easily generate reports showing your improvement progress using the chosen metrics for audit or governance purposes.

Key takeaways

• Clear, well-defined metrics are essential for effective healthcare improvement.
• Focus on validity, reliability, utility, feasibility, and alignment when selecting measures.
• Prioritise a 'vital few' measures: typically 1-2 outcome, 2-3 process, and 1-2 balancing.
• Always operationally define each metric with clear numerators, denominators, and data sources.
• Involve front-line staff and patients in the metric selection process.
• Regularly review and refine your metrics to ensure they remain relevant and useful for your project.