Public reporting

The public release of healthcare outcome or quality metrics is becoming increasingly common throughout the world. As healthcare continues its move towards pay for performance with demand-driven competition, the public reporting of quality, safety, and performance data may drive improvement by providing an incentive to produce change.28 It has been suggested that this driving force occurs via three separate pathways: (1) selection, where healthcare providers strive to improve quality so that more patients select that provider for their care; (2) change, where healthcare providers recognize deficiencies in their metrics and are motivated to improve care protocols to optimize their patient care; and (3) reputation, where the reputation of poorly performing providers is damaged through their publicly reported outcomes and therefore attempts are made to improve their reputation in the community.29 30

Public reporting of healthcare metrics provides transparency that allows patients to make informed decisions about where they receive care and allows local policy makers to better design regional stroke care networks, by diverting patients to high-performing centers and allocating additional resources to improve underperforming centers. With selection of providers by informed patients, high-performing centers will thus be rewarded and poor performers will be avoided. The degree to which public reporting can affect patient perspectives has been examined in the cardiovascular literature and shown to be considerable. In one study, >60% of patients viewed publicly available mortality data as accurate, useful, and likely to improve care, while <30% of the cardiovascular experts interviewed agreed.31

Controversially, public reporting may allow for dissemination of provider performance data and could influence reimbursement in a value-based healthcare system. Thus underperformers might be adversely affected by patient-informed selection and also by financial penalties by third-party payers. Preliminary data examining value-based purchasing models have even suggested potential detrimental effects to quality of care, as underperformers receive less reimbursement and are unable to invest in improvements, potentially worsening the quality of care. These are highly contentious concerns, but they are likely very relevant to the evolving medical marketplace.32 33

Essential elements of publicly reported data

For public reporting to improve quality, the disseminated data must be valid, reliable, and useful. The Society for Cardiac Angiography and Interventions position statement on public reporting identifies four essential requirements: the data must (1) “address the consumer’s important questions and priorities; (2) present credible information that is interpretable by the consumer; (3) stimulate the consumer to act on the information; and (4) avoid generating false perceptions regarding individual and/or institutional quality of care.”34 It is clear that data that fail to meet these criteria will have limited effect on producing beneficial change and may result in harm to providers that are disparaged by misleading or invalid data.

The cardiovascular surgery experience

The specialty with the most experience in public reporting is cardiovascular surgery. Modern public reporting systems began in New York State in the late 1980s with coronary artery bypass grafting report cards, which resulted in dramatic reductions in risk-adjusted operative mortality.41 Since then, public reporting of metrics has expanded to include other procedures within the specialty—most notably, percutaneous coronary intervention (PCI). By 2016, four states had active mandatory public reporting of PCI data (New York, Massachusetts, Texas and Washington) while other states (Pennsylvania and California, for instance) had limited systems.34 In the state of New York, data are reported by hospital and also individual providers; other mandatory reporting states report only hospital data. While the data that are captured differ by state, the cornerstone in PCI is the 30-day risk-adjusted mortality rate (RAMR), which is usually calculated by dividing the observed in-hospital 30-day mortality rate by the expected rate. Nationally, RAMR for PCI is approximately 1.3%. The RAMR is intended to control for patient comorbidities and disease severity to put providers who manage patients of different acuity on a level playing field.

However, RAMR has a number of potential limitations as a primary metric for institutional or proceduralist PCI performance. Most notably, death after PCI often depends more on the acuity of the presentation and comorbidities than on the technical prowess or judgement of the operator.34 Many argue that risk adjustment is particularly imprecise when dealing with the sickest patients who have the highest risk, leading to a perception of poor quality when a provider may merely have a comparatively high-risk case mix. For instance, a survey of cardiologists by Narins et al in 2005 showed that 86% of respondents did not agree that risk-adjustment models were adequate to control for high-risk patients.42 Further, the low national RAMR makes differentiation between centers difficult, as very large numbers of patients are needed to narrow confidence intervals to allow for detection of a significant difference. This means that data on true differences in performance often cannot be interpreted. Additionally, RAMR focuses on mortality when PCI was performed, but does not capture mortality for patients when PCI was not performed, and therefore reported metrics fail to consider the impact of interventionalist risk avoidance behavior on poor outcomes.

Since high-risk patient populations may skew publicly reported metrics, there are now efforts to exclude high-risk patients endorsed by major medical organizations, including the American Heart Association, Society for Cardiovascular Angiography and Interventions, and the Board of Governors of the American College of Cardiology.34 43 44 These patients include those with out-of-hospital cardiac arrest, acute cardiogenic shock, or hypoxic brain injury. New York State, for instance, now excludes these three patient groups from RAMR reporting.

PCI public reporting and risk avoidance behavior

One of the most important consequences of the public reporting of data is the relationship to risk-averse medical practice. With PCI, data from states with reporting indicate that centers may develop risk-avoidance behaviors, where treatment is withheld from higher-risk patients to avoid increasing reported mortality rates. A Medicare analysis of 100 000 patients with acute myocardial infarction in 10 states showed that 12% fewer PCI procedures were performed in states with public reporting, disproportionately affecting high-risk patients. This study also showed that the implementation of public reporting in Massachusetts led to a statistically significant lower PCI rate in acute myocardial infarction than in a cohort of non-reporting states.45 Two studies comparing New York PCI patients and outcomes with those in Michigan (without mandatory reporting) showed that high-risk patients were significantly less likely to undergo treatment in New York than in Michigan.46 47 Other studies have suggested similar findings.48 49 This is true also in the 2005 Narins survey, in which approximately 80% of New York interventionalists indicated that public reporting had influenced their decisions about whether to pursue PCI in certain patients.42 This indicates that providers may make treatment decisions based on public reporting and therefore may withhold life-saving treatment from patients who may need it the most to avoid negative scoring. Measures to mitigate this bias, such as removal of patients with cardiac arrest or cardiogenic shock from RAMR calculations, have led to a reduction in self-reported risk avoidance.31 Further, disease-specific reporting (including all patients with a specific diagnosis, regardless of treatment) as opposed to procedure-specific reporting (for instance, only patients undergoing PCI) may be integral in reducing the influence of public reporting on medical decision-making.

Public reporting for stroke

With advancements in systems of stroke care and with evidence now overwhelmingly supporting NT in the setting of ELVO, public reporting of outcome measures needs to be strongly considered as a ‘next step’ in advancing stroke care in the United States. The available data suggest that public reporting will probably have a positive effect on the outcomes and mortality of patients with stroke. However, the data also suggest that public reporting may lead to risk-avoidance behavior in choosing patients for NT and may inappropriately damage the reputation of institutions with higher-risk case mixes. Therefore, the decision-making process for public reporting should consider: how to adjust accurately for risk, whether to avoid clinical outcome metrics and present only process metrics (thereby minimizing risk avoidance), and whether to include disease-specific versus procedure-specific metrics.

Importantly, public reporting of stroke metrics should not be difficult and expensive for centers. All comprehensive stroke centers already track relevant metrics as part of their certification requirements (table 1), though not all these metrics would be practical or reliable. For instance, if public reporting of Thrombolysis in Cerebral Infarction reperfusion grade were implemented, there would probably be a relative increase in the percentage of self-reported good reperfusion grades because as there is no mechanism for external validation of scores there would be a bias towards scoring more highly. Additionally, patients with stroke are probably more heterogeneous than patients undergoing PCI, resulting in a wide spectrum of stroke clinical syndromes making comparison difficult, even when stroke scores, comorbidities, and age are considered. For instance, an important marker of candidacy for thrombectomy is the amount of core infarct relative to penumbra, and there is no clear consensus on how best to measure or report this finding. Additionally, the wide spectrum of neurological deficits, causes of stroke, and vessels involved makes standardized outcome metrics difficult to compare, particularly when volumes are low. A recent study of 10 stroke intervention centers (eight comprehensive, two primary) showed that over a given year, the average hospital capable of intervention performs only 76 NT procedures for stroke each year.50 A small number of thrombectomy procedures with which to establish outcome metrics, together with high patient heterogeneity and potentially inadequate risk adjustment, will probably result in misleading or unreliable data.

Potential stroke quality measures

Given the heterogeneous nature of patients presenting with stroke and the potentially deleterious effects of risk avoidance in stroke intervention, systems-based or procedural metrics may be a reasonable starting point for public reporting. Total procedural volume, door to arterial puncture, time to international normalized ratio reversal, and time to clot engagement are less dependent on patient-specific factors and reflect core institutional processes. Additionally, assessing overall improvement from presentation to discharge modified Rankin Scale score for all patients admitted with diagnosis of ischemic stroke, regardless of intervention, is a reasonable option. At the primary stroke center level, basic metrics of triage could be collected and reported, such as door-in to door-out times for patients progressing to a higher level of care. Reporting of these factors could be in stages, with the first stage reflecting private reporting of collected numbers for the purposes of institutional improvement, with an eventual roll-out to public reporting. Further, the initial metrics recorded could become more granular with time as the delivery of stroke care improves and more broad metrics are consistently met and exceeded. Currently, there are limited defined time target recommendations—for example, door to groin puncture or door-in to door-out times, to help stroke centers to guide patient care. Consensus from the stroke community on time goal targets for each metric will be necessary for implementing publicly reported standards but will also be useful in serving as guidelines for the community through which internal processes can be monitored and improved.