Article Text
PDF
Statistics from Altmetric.com
Introduction
The Patient-Centered Outcomes Research Institute (PCORI) was established by the Patient Protection and Affordable Care Act (ACA) of 2010 as a major new center to evaluate and implement clinical comparative effectiveness research (CER). It is a private, non-profit, tax exempt corporation designed to “assist patients, clinicians, purchasers and policy makers in making informed health decisions by advancing the quality and relevance of evidence concerning the manner in which diseases, disorders and other health conditions can effectively and appropriately be prevented, diagnosed, treated, monitored and managed through research and evidence synthesis”.1 2 The ACA's directive is that the institute “shall enter into contracts for the management of funding and conduct of research with government agencies and academic or private sector research entities and that it shall give preference to the Agency for Healthcare Research and Quality (AHRQ) and the National Institutes of Health (NIH)”.
The ACA established two independent and independently funded boards to control medical costs. The Independent Payment Advisory Board task is to implement target growth rates for Medicare while the PCORI's task is to evaluate and apply CER.1 3–5 The intent underlying the formation of the two panels is that they will, with their commingled effects on what care can be provided and what will be paid, provide high quality care at a reasonable cost.
The supporters of the PCORI and ACA proclaim the advantages of the PCORI and CER.4 6–8 In a 2008 report exploring ways to reduce healthcare costs, the Congressional Budget Office wrote that CER would reduce total spending on healthcare in the USA by an estimated US$8 billion from 2010 to 2019 without consideration of regulatory burden.9 That figure seems impressive until one realizes that it is less than one-tenth of 1% of overall healthcare cost.
In contrast with the supporters,3 opponents state that CER is not a medical science but a political science and will be detrimental to healthcare. They cite the potential dangers of cost emphasis and centralization which may be used by payers in a fashion uncomfortable to patients and their providers.
Evolution of the concept of comparative effectiveness research
Despite the controversy surrounding comparative effectiveness, one can argue that CER has become the most celebrated research initiative in medicine in the USA. The intellectual roots of CER can be traced back to mid-18th century Scotland and ‘arithmetical medicine’, practiced by the graduates of Edinburgh Medical School.10–12 The first comparative effectiveness study, in fact, was initiated by James Lind who undertook a controlled trial of six separate treatments for scurvy.12 In the USA, the first known use of CER has been attributed to Ernest Codman, at the beginning of the 20th century, with his implementation of ‘outcomes management’ in patient care.13
The formal evaluation and application of CER in the USA can be traced to 1989 when the Agency for Healthcare Policy and Research was created as an arm of the Department for Health and Human Services.14 The Agency for Healthcare Policy and Research undertook a number of initiatives, including the creation of the National Guideline Clearinghouse, designed to summarize the available medical evidence on the appropriate treatments for various conditions.14 It has been reported that they produced 15 guidelines at a total cost of US$750 million. In the mid-1990s, controversy arising as a result of some of these agency sponsored guidelines led to efforts to eliminate the agency.14
In 2003, the landmark Medicare Modernization Act (MMA) authorized the AHRQ to spend up to US$50 million the following year and additional amounts in future years to conduct and support research with a focus on “outcomes, comparative clinical effectiveness and appropriateness of healthcare items and services” for Medicare and Medicaid enrollees.15 16 The AHRQ has established an ‘effective healthcare’ program consisting of three main functions: reviewing and synthesizing existing evidence (using its evidence based practice centers); generating new information using a set of approved research centers (such as the HMO research network) that have access to data from medical claims and electronic medical records; and publishing findings and formats that are geared to the differing needs of clinicians, patients and policy makers.14
On February 17, 2009, President Obama signed into law the historic American Recovery and Reinvestment Act (ARRA). Of US$787 billion that was appropriated, US$150 billion was allotted for medical issues, with US$1.1 billion to CER.17 The ARRA also established the Federal Coordinating Council of Comparative Effectiveness Research.
The ACA of 20101 2 established the PCORI, a non-profit corporation charged with identifying research priorities, developing a research project agenda and executing research to better inform patients and physicians on their treatment choices. Recently, the Institute of Medicine (IOM) has recommended6 18 that the PCORI should be pivotal in providing future centralized CER, effectively diminishing or potentially eliminating the research conducted by individual clinicians, with a focus on public and private partnerships. In this context, private partnerships are generally the contractors, or occasionally academic organizations.
Patient-Centered Outcomes Research Institute
The basic framework has been developed by the Federal Coordinating Council of Comparative Effectiveness Research established through the ARRA. The ARRA established the Federal Coordinating Council for CER to foster optimal coordination of CER conducted, supported by the federal government. The council consists of 15 members, all of whom must be government employees and at least half of whom must have clinical experience. Writing on the structure of the Federal Coordinating Council of Comparative Effectiveness Research, Ezekiel Emanuel,19 the brother of former White House Chief of Staff, Rahm Emanuel, and a member of the council, in his book Healthcare guaranteed, wrote, “the Institute for Technology and Outcomes Assessment will provide information on the effectiveness and cost to eliminate tests or treatments of marginal or no value”.
Framework of CER and PCORI
Since the final development of the PCORI is vested in its origins in the Agency for Health Care Policy and Research (AHCPR) and AHRQ, as modified by the MMA, with an intermediary boost from the ARRA, many of the elements are common for all three regulations, including stakeholder involvement, transparency, public participation and open decision making.1 2 20 These common elements, which we believe to be important components for any future success of PCORI or like initiatives, are recurrent themes throughout this manuscript.
Stakeholder involvement
The stakeholder involvement is described in all three laws, the MMA, ARRA and ACA. Table 1 illustrates the stakeholder involvement.1 2 20 While there are similarities, as illustrated in table 1, there are also differences.
Framework for conducting comparative effectiveness reviews: stakeholder involvement
Transparency
Although some elements of transparency has been established with the MMA, ARRA and ACA (table 2), multiple regulations written for the MMA related to AHRQ programs and PCORI continue to emerge.1 2 20
Framework for conducting comparative effectiveness reviews: transparency
Public participation
Public participation is considered pivotal in healthcare reform. However, the ACA, in its debate throughout and various regulatory initiatives often did not seem to have taken public participation as a critical issue at its core. We believe that this represents an unfortunate shortfall in the development of the ACA because, in our opinion, true public participation is not only important but mandatory.1 2 Table 3 illustrates the differences between the MMA, ARRA and PCORI on various aspects of public participation. As expected, public participation is more significant with the PCORI, even though it may still be inadequate and impractical.1 2 20
Framework for conducting comparative effectiveness reviews: public participation
Open decision making
It is the opinion of the authors that open decision making should not be considered optional. However, it appears thus far that many of the decisions have been made behind closed doors without sufficient true public input. Furthermore, when public input is required, it seems that it is often not taken into consideration.1 2 As illustrated in table 4, open decision making has been appropriated for all three legislations with significant variations among them.1 2 20
Framework for conducting comparative effectiveness reviews: open decision making
Issues related to PCORI and CER
The issues related to the PCORI and CER are intertwined. The process that the federal agencies and private sector organizations use to set research priorities is central to the benefit that patients, physicians and others will receive from the investment in this project by Congress. The authors have reservations regarding the proscribed regimen of oversight. Multiple priorities have been described by various organizations.1
Stakeholder involvement
The major element of an acceptable process is the inclusion of all interested and affected stakeholders.
The PCORI's main mechanism for stakeholder participation is through its oversight and/or advisory bodies, only one of which, the Board of Governors, deals explicitly with priority setting. The government Accountability Office announced the appointments to the 21 member board in September 2010. However, the literature emphasizing the importance of stakeholder engagement as well as the concerns that arose during the ARRA implementation suggests that a more robust engagement mechanism than the Board of Governors may be necessary.20
It appears that the PCORI program under the ACA has broad stakeholder participation; however, the internal composition itself has raised questions. The IOM and others are essentially silent on the constitution of the stakeholder groups. However, there is a white paper on that point, a National Strategy for CER Dissemination by the National Network for Health Innovation which does recommend that national medical societies and patient groups are crucial as potential partners going forward. The group consists of four physicians with at least one surgeon although there is no indication whether there is any requirement that they be currently practicing. The composition also provides significant influence for private payers with three representatives, three members representing pharmaceutical devices, etc, and three patient representatives. While broad participation is admirable, one wonders how such a group will actually achieve consensus.
Almost definitionally, successful dissemination of CER evidence will require collaboration. The highly decentralized nature of evidence dissemination, collaboration and consensus building is and will be critical, even when CER produces findings that seem to be clear and unambiguous. IOM is recommending a centralized process. In general, stakeholder groups that are not part of a consensus building process are less likely to become active advocates for the dissemination of guidelines or other evidence based aids to decision making or may in fact become active opponents of guidelines.
Thus national medical societies and patient groups could be potential and crucial partners for the PCORI. Many national organizations operate programs to identify and disseminate standards of practice; however, it has been suggested that few have specifically focused on the development of mechanisms to build rapid consensus and expedite the use of good evidence. Furthermore, they also claim that limited funding and the complexity of bringing experts together can constrain the consensus building process, as evidenced by the NIH Consensus Development Program, which releases no more than three or four consensus statements per year. However, there is no proof for such allegations and there is also no evidence that anything developed by the NIH Consensus Development Program has been methodologically sound, or improved healthcare, access to the patients or even costs.
Evidence based medicine and CER are based not only on methodologic quality assessment of studies but also on the understanding of the clinical nature of the technique, procedure or intervention being evaluated.21–23 Overall, as in the case of the PCORI, stakeholder groups may disagree on the implications of evidence, even when the evidence seems to be clear. Furthermore, many of the groups which are contracted outside the agency may possess substantial conflicts of interest, as the income source even in non-profit organization can be unclear and as representatives from various insurance and manufacturer stakeholders may have conflicting loyalties. Recently, the American College of Chest Physicians and the American Academy of Orthopedic Surgeons took markedly different positions on the outcomes to be used in defining the efficacy of measures to prevent thromboembolism during hip and knee surgery.24 The authors believe that similar challenges apply to interventional pain techniques and surgical interventions. There is often disagreement between interventional pain physicians and surgeons with resultant contradictory opinions.
The conflict in guideline preparation dissemination, transparency and openness has been well described, along with intricacies, complications and difficulties in the systematic assessment of evidence.10 11 23–27
We believe that the PCORI should challenge all the communities in medicine to rally around a common approach to evidence synthesis, consensus building and the dissemination of appropriate guidelines with proper stakeholder involvement.
Transparency
The process must be transparent and open. It is a key element throughout the process. Limited transparency may limit the value of each of the subsequent elements.
Transparency is an important issue not only for PCORI but at many levels of healthcare. We believe the PCORI should take the lead and publish transparently all the stakeholders involved and their conflicts of interest and provide relevant financial information in great detail. The PCORI is also best suited to provide transparency in their own knowledge gaps or other challenges they face. Furthermore, the Institute's reports should not only go to Congress and to the President, they should also be made available to the public, including details of the institute's research methodology priorities, activities and conflicts of interest.
Public participation
Stakeholder involvement, transparency, public participation and open decision making all go hand in hand. Public participation should never be a nominal or a philosophical approach but it should be substantive with consideration of all aspects, including the ability to admit the shortcomings of the commission. In the past, the public forums and listening sessions were offered by the IOM and the Federal Coordinating Council for CER. The PCORI has a 45–60 day comment period for the public to respond to research priorities, and also requires the Institute to provide additional input opportunities (eg, through a website). This approach still might not ensure meaningful and comprehensive input, and this might be considered deficient.
Open decision making
The decision making must be open not only by vote of the governors but also by the ad hoc committees. CER decisions to date have been made by federal officials, with no opportunity for the public to witness how important determinations are made, how research priorities are set, how funding decisions are made, how research gaps and methodological challenges are considered and addressed, and how non-public meetings with particular stakeholders impact decisions. Finally, the cost of any of these activities is never provided to the public or, for that matter, to Congress. Further, the involvement of private contractors must be open and transparent showing their own conflicts of interest. Consequently, a more open decision making process might enhance the credibility of these determinations and increase its acceptability.
Open decision making is particularly essential for the methodological process. The expert advisory panel, including patient subgroups, should provide advice on research designs and protocols, and it requires that a methodologic committee develop standards for conducting CER that account for and evaluate patient subpopulations. As an example, in interventional pain management, deficiencies exist at the present time in understanding randomized control trials, their subgroups, and placebo control and nocebo hyperalgesia. With this subspecialty as well as others, attention must be focused on effectiveness rather than efficacy and, as much as possible, away from strict adherence to academic centers and placebo controlled randomized double blind trials, of which there are a limited number in interventional pain management.
Finally, in any given area the institute must follow the same rules and regulations for all interventions considered in the research.
PCORI and CER in the USA
CER in the USA is confused with evidence based medicine. Although similar, there are important differences. Previous efforts have often failed to utilize appropriate methodological principles and lack accountability and transparency. Most organizations only discuss the process and seem to inadequately state their deficiencies and conflicts of interest.
Multiple organizations, either for profit or not for profit, are vested in CER. Even though these organizations may appear as not for profit, they do have a business to run, including payroll and maintenance. Conflicts of interest range from the origin of the funding to the final use. Funding may come from manufacturers or insurers with divergent philosophies and agendas. Authors of CER documents and guidelines regularly fail to disclose their incomes and how they have been derived, which creates an appearance of a conflict of interest. As an example, the Washington State Health Authority System has produced numerous guidelines, some entirely without public input. Their recent spinal injections guidelines were evaluated by Spectrum Research by only one physician. This process produced controversial guidelines, without information on the cost and benefits derived by the company and the participants.28 It seems that none of the peer review suggestions was taken into consideration.29 Furthermore, it appeared to this observer that the open discussion forums were avoided although they were ultimately heard with substantial restrictions. The final decision by the committee took into consideration only what was presented by Spectrum Research, which was flawed and, in our opinion, quite biased, as well as some individual presentations. Understandably, the committee members neither have time nor resources to review the evidence themselves; however, we believe that they should at least have reviewed the peer review reports and the reports from others. Of note, one of the members of this organization is also on the Board of Governors of the PCORI, even though this individual's expertise is related to administration.
Numerous divergent guidelines and interpretations of the same evidence with differing results by different groups have been presented. This has been frequently illustrated by multiple authors who are negative about certain intervention.1 10 11 22 23 25 28 29
Conclusion
The PCORI should not only outline the true principles of transparency, independence, and stakeholder involvement but also non-discrimination of the evidence. The ACA is both historic and transformational. It also remains troubling for some and controversial for many. There is lack of understanding in the public about what this law does and its effects. It appears that it will result in insurance for many more people but potentially decreased coverage as well.
PCORI could be good for promoting patient centered physician behavior. The potential harm from the PCORI depends on how the research is used, which could easily quell medical innovation by centralizing care. The fear is that officials seeking to control costs would use this research to restrict access to more costly medical interventions, as done by the National Institute for Health and Clinical Excellence. As the government puts more people into the US healthcare system and the promised savings fail to materialize, there will be tremendous pressure on elected officials to slow spiraling costs, which can reduce access and quality.
The supporters are starting to promote CER and its mission, that the PCORI must take a visible leadership role and make bold decisions. However, the skeptics and opponents have a different view of PCORI operations. We the authors look forward to further clarification of the process and hope it eventually lives up to the promise of promoting improved health care through policies implemented based upon appropriately designed and executed patient centered outcomes research.
Acknowledgments
The authors thank Pain Physician for providing permission to publish in an abbreviated form.
Footnotes
The manuscript is a brief version of Manchikanti L, et al The impact of comparative effectiveness research on interventional pain management: evolution from Medicare Modernization Act to Patient Protection and Affordable Care Act and the Patient-Centered Outcomes Research Institute. Pain Physician 2011;14:E249–282. This version is published with the consent of all of the authors and the permission of the journal Pain Physician.
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.