Pitfalls in measuring healthcare outcomes
Introduction
You might not have heard of Goodhart's Law, but it describes an extremely common tendency that has almost certainly affected us all at some point.
Goodhart's Law states, "When a measure becomes a target, it ceases to be a good measure."
In simpler terms, once a specific metric is used as the main indicator of success, its reliability as an accurate measure tends to decrease. This can happen because people start manipulating or gaming the system to achieve high scores on that metric, often at the expense of broader, more useful objectives. A more positive (less cynical) way of expressing this is that having the wrong criteria for success will mean that many hardworking people are working very hard on the wrong things.
In a healthcare context, the UK government has often talked about "target waiting times" for things like waiting times in Accident and Emergency, or the time spent waiting to undergo an elective surgical procedure. Such "waiting times" have the perceived benefit of being easy to communicate by politicians and understandable by patients. The problem is that this metric is not necessarily what it is made out to be - a reflection of system quality. What truly matters is not usually that which is most easily measured. If we fail to keep that in mind, we run the risk of becoming good at things that don't really matter.
For example, evidence suggests that "target waiting times" for services such as those in accident and emergency or elective surgical procedures have been manipulated to make the NHS system performance appear better than it is. This manipulation of waiting times has been a concern, as highlighted in several instances:
Cancer Charity's Concerns: A report from Radiotherapy UK questioned NHS England's claims about improving waiting times, suggesting that cancer targets had been manipulated to gloss over a worsening situation. (O'Dowd, 2023)
Elective Care Waiting Times: A report by the Parliamentary Public Accounts Committee indicated that a quarter of reported waiting times for elective care in hospitals in England were probably incorrect. The report highlighted that the use of inconsistent systems across trusts made it challenging to make accurate comparisons and could mislead patients about their choices for treatment locations (O'Dowd, 2014)
General Manipulation of Data: The manipulation of waiting times has been a broader issue, not just limited to specific services. For example, the historical manipulation of waiting times data to meet targets has been a recurring theme, suggesting systemic issues within the NHS that affect the reliability of reported performance metrics (Kay, 2019)
These instances indicate a pattern where the pressure to meet predefined targets can lead to practices compromising the accuracy and integrity of data reporting within the NHS. This manipulation misrepresents the system's performance and potentially impacts patient care and public trust in the healthcare system.
Relevance to Physical Rehabilitation
Goodhart's Law can have significant implications in the context of physical rehabilitation. Rehabilitation is, by its nature, a challenging domain. I have never met anyone who complained they received too much rehabilitation following their stroke or spinal cord injury. It can be very difficult to set goals for rehabilitation and strike the right balance between restitution and compensation. The healthcare system must almost certainly treat many people with constrained resources.
If a rehabilitation centre focuses solely on the number of patients it treats in a given period as a measure of success, the quality of care might decline. Therapists might need to limit the number or duration of sessions to increase patient turnover, potentially leading to inadequate treatment and poorer patient outcomes. Similarly, suppose a metric like "patient satisfaction scores" becomes a target. In that case, the centre might implement superficial changes that improve these scores without genuinely enhancing the quality of care or patient recovery outcomes.
To mitigate the effects of Goodhart's Law in physical rehabilitation, it's crucial to use a balanced set of metrics that reflect both the quantity and quality of care. These might be more difficult to communicate to the public but at least they should be more reflective of the quality of treatment episodes. This approach helps ensure that improving one metric doesn't detrimentally affect other important aspects of patient care and rehabilitation outcomes.
Adopting a comprehensive approach that prioritises quality of care and patient outcomes above basic metrics is crucial to circumvent the drawbacks associated with Goodhart's Law. Below are several strategies to consider:
Use a Balanced Set of Metrics
Instead of relying on a single indicator, such as the number of patients treated, rehabilitation centres should use a balanced set of metrics assessing various care aspects. These could include patient satisfaction, improvement in functional capabilities, and long-term recovery rates. This approach helps ensure that improving one metric does not negatively impact other crucial aspects of care.
Focus on Patient-Centered Outcomes
Shift the focus from quantitative metrics to patient-centered outcomes. This involves tracking patients' progress toward their specific goals and quality of life improvements. For example, measuring the extent to which patients regain mobility or experience a reduction in pain post-rehabilitation can provide a more accurate picture of the treatment's effectiveness.
Implement Qualitative Feedback Mechanisms
Integrating patient and staff feedback is crucial for qualitatively assessing the rehabilitation process. Regular surveys, interviews, and focus groups offer a depth of insight that quantitative data alone cannot. This invaluable feedback highlights areas for enhancement that might not be evident through primary metrics alone.
Continuous Training and Education
Ensure that staff are trained to meet the metrics and educated on the importance of holistic patient care. Continuous professional development should be encouraged, focusing on the latest research and techniques in rehabilitation that promote comprehensive patient care.
Utilise Technology Appropriately
Technology is often portrayed as being "the answer" to rehabilitation effectiveness. However, this is not automatically the case. Sometimes we can leverage technology to enhance the quality of care without making it the sole focus. For instance, adopting telerehabilitation to supplement traditional methods could provide flexibility and continuity of care as patients go home, especially in challenging situations, as we discovered during the COVID-19 pandemic. However, it's crucial to ensure that technology does not compromise treatment quality (Lienou, 2021). Technology can also enhance the prospects for neuroplasticity by enabling many more repetitions of an exercise in a fixed session time or even enable movements not possible with any other intervention.
Regular Review and Adaptation of Practices
Review and adapt rehabilitation practices regularly based on outcomes and feedback. This dynamic approach allows for modifying strategies that do not yield the desired results, thus maintaining the integrity of the rehabilitation goals.
By implementing these strategies, physical rehabilitation centres can avoid the negative consequences of Goodhart's Law, ensuring that metrics serve as helpful guides rather than definitive targets. This balanced approach promotes genuine improvements in patient care and rehabilitation outcomes.
Conclusion
A narrow focus on quantitative metrics can lead to unintended consequences, including the degradation of care quality and patient outcomes. The phenomenon of Goodhart's Law, where "When a measure becomes a target, it ceases to be a good measure", clearly manifests in the healthcare sector.
The article underscores the importance of adopting a balanced set of metrics encompassing both the quantity and quality of care.
Strategies such as focusing on patient-centered outcomes, integrating qualitative feedback mechanisms, continuous staff training, appropriate technology utilisation, and regular review and adaptation of practices have been outlined as essential steps toward ensuring rehabilitation services meet their primary goal of enhancing patient recovery and quality of life.
Achieving this balance is paramount for the integrity of rehabilitation practices and for maintaining public trust in the healthcare system.
References
O'Dowd A. "Cancer charity questions NHS England's claim that waiting times are improving." BMJ. 2023 Sep 4;382:2028. doi: 10.1136/bmj.p2028. PMID: 37666520. https://pubmed.ncbi.nlm.nih.gov/37666520/
O'Dowd A. "A quarter of reported waiting times for elective care are probably wrong, MPs say." BMJ. 2014 Apr 29;348:g3013. doi: 10.1136/bmj.g3013. PMID: 24780310.
Kay, Adrian. “The NHS’s Assault on Waiting Times.” (2019).
https://www.semanticscholar.org/paper/The-NHS’s-Assault-on-Waiting-Times-Kay/82186304f091a93b0ca319e573101d1ff8fa3d1e
Lienou, Thierry. “Telerehabilitation versus traditional rehabilitation of adults with upper-extremity orthopedic conditions: a systematic review.” (2021). https://www.semanticscholar.org/paper/Telerehabilitation-versus-traditional-of-adults-a-Lienou/503e1a6423bdfaed97569688c412a27f8e9c26b8