Public reporting of healthcare metrics provides transparency that allows patients and emergency medical providers to make informed decisions about where patients should receive care. Most previous reports about public reporting of health metrics have demonstrated significant improvements in outcome metrics after implementation. However, no mechanism exists, voluntary or otherwise, for the public reporting of outcomes of stroke care. We review the components of public reporting of health outcomes data and its limited history in stroke outcomes. We summarize the literature on public reporting in cardiovascular interventions, particularly percutaneous coronary interventions, as a close corollary to mechanical neurothrombectomy. The benefits, limitations, and controversies associated with reporting of cardiovascular outcomes are reviewed with a focus on the development of risk-avoidant behaviors. This article serves as a primer for discussion of the potential benefits, limitations, and unintended consequences of public reporting of stroke data.
Statistics from Altmetric.com
During the past 3 years, multiple randomized controlled trials have demonstrated a strong benefit of neurothrombectomy (NT) for emergent large vessel occlusion (ELVO).1–7 Accordingly, in 2015 the American Heart Association/American Stroke Association (AHA/ASA) provided a class 1, level of evidence A recommendation that NT be performed in patients with ELVO meeting appropriate criteria.8 Further studies have suggested that approximately 10 000 NT procedures were performed in the United States during 2015, and this number is expected to increase.9–13 Recently, the DAWN and DEFUSE trials showed strong superiority of NT over medical management for patients presenting with favorable perfusion imaging 6–24 hours after symptom onset.14 15 In light of these results and the 2018 AHA/ASA guidelines, thrombectomy is rapidly becoming the standard of care for patients with stroke with delayed presentations and preserved penumbra.16 17 These results will almost certainly further increase the indications for NT, with a corresponding increase in procedural volumes.
As of 2017, there are more than 1000 Joint Commission-certified comprehensive stroke centers and primary stroke centers across the United States. The literature indicates a strong relationship between higher-volume stroke centers and better clinical outcomes.18–24 The Joint Commission has recently proposed a new certification for hospitals—a thrombectomy-capable stroke center (TSC) advanced certification—intended as an intermediate classification between these two accredidations.25 The impact of TSCs remains to be seen, with the potential to extend ischemic stroke care; however, this may come at the cost of suboptimal regionalization of care and dilution of procedural volume among too many centers.26
Although advanced stroke certification requires that hospitals meet rigorous professional, procedural, and clinical standards; requirements are unclear for clinical volume, process, or reporting of outcome data, with no mechanism for voluntary public reporting. Consequently, no data are available to compare even basic process outcomes across different stroke centers or to allow comment on minimum-acceptable metrics for continued accreditation. For example, a recent study by Chaturvedi and Loftus queried the ability of the Medicare Hospital Compare website, which compared 30 hospitals in a single metropolitan area, to assess the quality of publicly reported data on carotid endarterectomy (CEA) and carotid angioplasty and stenting (CAS) between 2011 and 2012. No data were available for CAS, only 27% of hospitals provided detailed CEA volumes, and no hospitals provided periprocedural CEA stroke or death rates.27 Therefore, emergency medical service providers, patients, and families have no information to help them choose which centers they should go to for treatment of a stroke. Most importantly, the absence of a public reporting mechanism means that continually underperforming stroke centers may continue to subject patients to unnecessarily poor outcomes without the patients knowing that other centers might achieve better results. With clear, class I evidence supporting the benefit of NT in patients with stroke presenting with ELVO, it is important to re-examine the potential value of public reporting of stroke center metrics.
The public release of healthcare outcome or quality metrics is becoming increasingly common throughout the world. As healthcare continues its move towards pay for performance with demand-driven competition, the public reporting of quality, safety, and performance data may drive improvement by providing an incentive to produce change.28 It has been suggested that this driving force occurs via three separate pathways: (1) selection, where healthcare providers strive to improve quality so that more patients select that provider for their care; (2) change, where healthcare providers recognize deficiencies in their metrics and are motivated to improve care protocols to optimize their patient care; and (3) reputation, where the reputation of poorly performing providers is damaged through their publicly reported outcomes and therefore attempts are made to improve their reputation in the community.29 30
Public reporting of healthcare metrics provides transparency that allows patients to make informed decisions about where they receive care and allows local policy makers to better design regional stroke care networks, by diverting patients to high-performing centers and allocating additional resources to improve underperforming centers. With selection of providers by informed patients, high-performing centers will thus be rewarded and poor performers will be avoided. The degree to which public reporting can affect patient perspectives has been examined in the cardiovascular literature and shown to be considerable. In one study, >60% of patients viewed publicly available mortality data as accurate, useful, and likely to improve care, while <30% of the cardiovascular experts interviewed agreed.31
Controversially, public reporting may allow for dissemination of provider performance data and could influence reimbursement in a value-based healthcare system. Thus underperformers might be adversely affected by patient-informed selection and also by financial penalties by third-party payers. Preliminary data examining value-based purchasing models have even suggested potential detrimental effects to quality of care, as underperformers receive less reimbursement and are unable to invest in improvements, potentially worsening the quality of care. These are highly contentious concerns, but they are likely very relevant to the evolving medical marketplace.32 33
Essential elements of publicly reported data
For public reporting to improve quality, the disseminated data must be valid, reliable, and useful. The Society for Cardiac Angiography and Interventions position statement on public reporting identifies four essential requirements: the data must (1) “address the consumer’s important questions and priorities; (2) present credible information that is interpretable by the consumer; (3) stimulate the consumer to act on the information; and (4) avoid generating false perceptions regarding individual and/or institutional quality of care.”34 It is clear that data that fail to meet these criteria will have limited effect on producing beneficial change and may result in harm to providers that are disparaged by misleading or invalid data.
Effect of public reporting on outcomes
A recent meta-analysis and systematic review published by Campanella and colleagues evaluated the impact of public reporting on mortality and clinical outcomes.35 The study included 27 manuscripts, many derived from cardiology literature for procedural outcomes, but others dealing with other clinical metrics such as readmissions, wait times, hip fractures, antibiotic use, and injection prescribing rates. Pooling the 10 manuscripts with sufficient data to evaluate mortality, the authors found a protective effect with a relative risk of mortality of 0.85 associated with public reporting (P<0.0001). Studies evaluating other clinical outcomes were heterogeneous and could not be analyzed statistically in a similar manner. However, of the 27 studies, 14 reported a positive effect of public reporting on outcomes, nine reported a statistically non-significant suggestion of benefit, three had mixed results, and one study suggested a deleterious effect. These findings were consistent with previous reviews of published data on public reporting.36 37 Overall, the authors suggest that most studies suggest a positive correlation between clinical outcomes and public reporting of data, although this relationship is not consistent or clearly understood.35
Public reporting of stroke metrics
Data evaluating the effect of public reporting on clinical metrics for stroke are limited. A 2012 systematic review by Parker and colleagues identified only two relevant studies for inclusion in the analysis, although both provided limited data on the relationship between public reporting and outcomes.38 The first study, by Kelly et al, suggested poor reliability of stroke outcome data reported through the Agency for Healthcare Research and Quality.39 The second, by Hollenbeak et al, suggested that patients with hemorrhagic or ischemic stroke treated at 200 Pennsylvania hospitals with mandatory reporting between 2000 and 2003 had significantly lower mortality (OR=0.57 and 0.77, respectively) than patients treated at 34 non-Pennsylvania hospitals with voluntary (non-mandatory) reporting requirements.40 Unfortunately, these hospital samples are almost certainly non-equivalent, which biases interpretation of these findings.
The cardiovascular surgery experience
The specialty with the most experience in public reporting is cardiovascular surgery. Modern public reporting systems began in New York State in the late 1980s with coronary artery bypass grafting report cards, which resulted in dramatic reductions in risk-adjusted operative mortality.41 Since then, public reporting of metrics has expanded to include other procedures within the specialty—most notably, percutaneous coronary intervention (PCI). By 2016, four states had active mandatory public reporting of PCI data (New York, Massachusetts, Texas and Washington) while other states (Pennsylvania and California, for instance) had limited systems.34 In the state of New York, data are reported by hospital and also individual providers; other mandatory reporting states report only hospital data. While the data that are captured differ by state, the cornerstone in PCI is the 30-day risk-adjusted mortality rate (RAMR), which is usually calculated by dividing the observed in-hospital 30-day mortality rate by the expected rate. Nationally, RAMR for PCI is approximately 1.3%. The RAMR is intended to control for patient comorbidities and disease severity to put providers who manage patients of different acuity on a level playing field.
However, RAMR has a number of potential limitations as a primary metric for institutional or proceduralist PCI performance. Most notably, death after PCI often depends more on the acuity of the presentation and comorbidities than on the technical prowess or judgement of the operator.34 Many argue that risk adjustment is particularly imprecise when dealing with the sickest patients who have the highest risk, leading to a perception of poor quality when a provider may merely have a comparatively high-risk case mix. For instance, a survey of cardiologists by Narins et al in 2005 showed that 86% of respondents did not agree that risk-adjustment models were adequate to control for high-risk patients.42 Further, the low national RAMR makes differentiation between centers difficult, as very large numbers of patients are needed to narrow confidence intervals to allow for detection of a significant difference. This means that data on true differences in performance often cannot be interpreted. Additionally, RAMR focuses on mortality when PCI was performed, but does not capture mortality for patients when PCI was not performed, and therefore reported metrics fail to consider the impact of interventionalist risk avoidance behavior on poor outcomes.
Since high-risk patient populations may skew publicly reported metrics, there are now efforts to exclude high-risk patients endorsed by major medical organizations, including the American Heart Association, Society for Cardiovascular Angiography and Interventions, and the Board of Governors of the American College of Cardiology.34 43 44 These patients include those with out-of-hospital cardiac arrest, acute cardiogenic shock, or hypoxic brain injury. New York State, for instance, now excludes these three patient groups from RAMR reporting.
PCI public reporting and risk avoidance behavior
One of the most important consequences of the public reporting of data is the relationship to risk-averse medical practice. With PCI, data from states with reporting indicate that centers may develop risk-avoidance behaviors, where treatment is withheld from higher-risk patients to avoid increasing reported mortality rates. A Medicare analysis of 100 000 patients with acute myocardial infarction in 10 states showed that 12% fewer PCI procedures were performed in states with public reporting, disproportionately affecting high-risk patients. This study also showed that the implementation of public reporting in Massachusetts led to a statistically significant lower PCI rate in acute myocardial infarction than in a cohort of non-reporting states.45 Two studies comparing New York PCI patients and outcomes with those in Michigan (without mandatory reporting) showed that high-risk patients were significantly less likely to undergo treatment in New York than in Michigan.46 47 Other studies have suggested similar findings.48 49 This is true also in the 2005 Narins survey, in which approximately 80% of New York interventionalists indicated that public reporting had influenced their decisions about whether to pursue PCI in certain patients.42 This indicates that providers may make treatment decisions based on public reporting and therefore may withhold life-saving treatment from patients who may need it the most to avoid negative scoring. Measures to mitigate this bias, such as removal of patients with cardiac arrest or cardiogenic shock from RAMR calculations, have led to a reduction in self-reported risk avoidance.31 Further, disease-specific reporting (including all patients with a specific diagnosis, regardless of treatment) as opposed to procedure-specific reporting (for instance, only patients undergoing PCI) may be integral in reducing the influence of public reporting on medical decision-making.
Public reporting for stroke
With advancements in systems of stroke care and with evidence now overwhelmingly supporting NT in the setting of ELVO, public reporting of outcome measures needs to be strongly considered as a ‘next step’ in advancing stroke care in the United States. The available data suggest that public reporting will probably have a positive effect on the outcomes and mortality of patients with stroke. However, the data also suggest that public reporting may lead to risk-avoidance behavior in choosing patients for NT and may inappropriately damage the reputation of institutions with higher-risk case mixes. Therefore, the decision-making process for public reporting should consider: how to adjust accurately for risk, whether to avoid clinical outcome metrics and present only process metrics (thereby minimizing risk avoidance), and whether to include disease-specific versus procedure-specific metrics.
Importantly, public reporting of stroke metrics should not be difficult and expensive for centers. All comprehensive stroke centers already track relevant metrics as part of their certification requirements (table 1), though not all these metrics would be practical or reliable. For instance, if public reporting of Thrombolysis in Cerebral Infarction reperfusion grade were implemented, there would probably be a relative increase in the percentage of self-reported good reperfusion grades because as there is no mechanism for external validation of scores there would be a bias towards scoring more highly. Additionally, patients with stroke are probably more heterogeneous than patients undergoing PCI, resulting in a wide spectrum of stroke clinical syndromes making comparison difficult, even when stroke scores, comorbidities, and age are considered. For instance, an important marker of candidacy for thrombectomy is the amount of core infarct relative to penumbra, and there is no clear consensus on how best to measure or report this finding. Additionally, the wide spectrum of neurological deficits, causes of stroke, and vessels involved makes standardized outcome metrics difficult to compare, particularly when volumes are low. A recent study of 10 stroke intervention centers (eight comprehensive, two primary) showed that over a given year, the average hospital capable of intervention performs only 76 NT procedures for stroke each year.50 A small number of thrombectomy procedures with which to establish outcome metrics, together with high patient heterogeneity and potentially inadequate risk adjustment, will probably result in misleading or unreliable data.
Potential stroke quality measures
Given the heterogeneous nature of patients presenting with stroke and the potentially deleterious effects of risk avoidance in stroke intervention, systems-based or procedural metrics may be a reasonable starting point for public reporting. Total procedural volume, door to arterial puncture, time to international normalized ratio reversal, and time to clot engagement are less dependent on patient-specific factors and reflect core institutional processes. Additionally, assessing overall improvement from presentation to discharge modified Rankin Scale score for all patients admitted with diagnosis of ischemic stroke, regardless of intervention, is a reasonable option. At the primary stroke center level, basic metrics of triage could be collected and reported, such as door-in to door-out times for patients progressing to a higher level of care. Reporting of these factors could be in stages, with the first stage reflecting private reporting of collected numbers for the purposes of institutional improvement, with an eventual roll-out to public reporting. Further, the initial metrics recorded could become more granular with time as the delivery of stroke care improves and more broad metrics are consistently met and exceeded. Currently, there are limited defined time target recommendations—for example, door to groin puncture or door-in to door-out times, to help stroke centers to guide patient care. Consensus from the stroke community on time goal targets for each metric will be necessary for implementing publicly reported standards but will also be useful in serving as guidelines for the community through which internal processes can be monitored and improved.
Neurothrombectomy is a time-sensitive and efficacious therapy. As a result, rapid pre-hospital triage of patients to high-quality intervention centers is paramount. Eventually, outcomes will be tracked across the spectrum of stroke care, leading to an overall improvement in the delivery of quality care. The potential difficulty lies in selecting what to report. As a community, we need to initiate this discussion and propose metrics which accurately reflect quality of care across severity of disease and access to care, before arbitrary measures are chosen for us.
Contributors KMF: conception, study design, data collection, manuscript preparation; JLW: data collection, manuscript preparation; JM: conception, study design, manuscript preparation.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests JM: consultant – Rebound Medical, EndoStream, Cerebrotech Medical, Viseon, Synchron; investor – Apama, Viseon. TSP, Vastrax, EndoStream, Synchron, Cerebrotech Medical, Neurvana, NTI, Serenity.
Patient consent Not required.
Provenance and peer review Not commissioned; externally peer reviewed.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.