The U.S Institute of Medicine (IOM) defines comparative effectiveness research (CER) as “the generation and synthesis of evidence that compares the benefits and harms of alternative methods to prevent, diagnose, treat, and monitor a clinical condition, or to improve the delivery of care.”1 The purpose of CER is to “assist consumers, clinicians, purchasers, and policy makers to make informed decisions that will improve health care at both the individual and population levels.”1 Two key elements are 1) the direct comparison of effective interventions, and 2) the application of these interventions in patients who are typical of day-to-day clinical practice.2

Although there has been a recent push toward adopting practical clinical trials that evaluate different medications or treatment algorithms in the context of everyday clinical practice,3 most of the medical literature remains focused on placebo-controlled trials, as they are commonly performed to obtain Food and Drug Administration (FDA) approval of a new treatment or intervention. They rarely address scenarios that are typical of day-to-day clinical practice, a key component of CER research. A systematic methodology for identifying and prioritizing CER research topics was recently published by the IOM. The process includes obtaining input from a broad array of “stakeholders,” not just physicians, to determine which study topics should be given the highest priority for funding. This set of top 100 national priorities of CER was published in 2009 and broadly spans all areas of medicine.1

The key research topic in the area of inflammatory bowel disease (IBD) from the IOM report was to compare the effectiveness of competing biologic agents in inflammatory diseases. Yet there are additional key questions that are important to study in the day-to-day care of persons with IBD not represented in the IOM report. To date, the IOM method has not been used to develop CER priorities relevant to the day-to-day clinical practice of a specific disease or specialty. The feasibility of this process outside of a large government-funded effort has also not been assessed. To this end, we adapted the methods developed by the IOM to determine the feasibility and establish a list of high-priority CER topics specifically related to the care of patients with IBD.

MATERIALS AND METHODS Sample and Survey Administration

Following the IOM's approach for developing CER priorities,1 we developed and administered a survey to practicing gastroenterologists asking them to suggest important CER topics in IBD (Supporting Table 1). The survey was electronically distributed to an international sample of gastroenterologists from the United States, Canada, the United Kingdom, and Australia using the membership lists of the American College of Gastroenterology (ACG), Canadian Association of Gastroenterology (n = 453), the Australian IBD Association (n = 128), and the British Society of Gastroenterology (n = 1470). To reach members of the ACG, an email was sent to approximately a 15% sample (n = 1650) of randomly selected physicians after protocol review and approval from the ACG. The other groups received the invitation to participate in the study as part of an emailed newsletter sent to members of the organization on the email distribution lists.

The gastroenterologists were requested to submit up to three free text response IBD CER topics that they felt were of the highest priority. For each proposed CER study, responders were asked to comment on 1) why each topic should be a national priority, 2) the primary and secondary area to be studied, and 3) the study population to be evaluated. We collected demographic information about the individual respondents, including age, number of years in practice, number of patients seen each month with IBD, percent of patients treated with IBD, and practice type. As an incentive to complete the survey, the gastroenterologists were notified they would be entered into a lottery for one of three Lenovo Netbooks to be awarded to randomly selected survey participants who completed the survey by the deadline.

To further bolster the proposed CER list, members of the Building Research in Inflammatory Bowel Disease Globally (BRIDGe) group independently proposed CER studies. This international group of 13 IBD specialists was formed to engage in IBD research (www.BRIDGe.com) and draws members from the United States, Canada, Australia, and the United Kingdom, and from both university and private practice settings. To ensure adequate representation of CER topics from various facets of IBD management, each BRIDGe member was assigned one or two categories of IBD care (Supporting Table 2) from which they were asked to submit a minimum of three CER topics. Including the required CER topics, each individual BRIDGe member could propose up to 20 individual CER topics from any category.

To invite input from other stakeholder groups, we convened two patient focus groups to understand the patients' perspective of the most important IBD research topics. The patient focus groups were held at Dartmouth-Hitchcock Medical Center (six patients) and in Los Angeles through the IBD Support Foundation (eight patients) and moderated by two members of the BRIDGe group (C.A.S. and G.Y.M.) following a scripted outline. We designed the interviews to be flexible so that patients felt free to offer their opinions. The script was prepared by one of the coauthors (G.Y.M.). After the introduction about CER and goals of the meeting, the focus groups began with an open-ended probe using the “think-aloud” technique of cognitive interviewing.4 The patients were instructed to suggest ideas for CER in IBD which could be related to prevention, diagnosis, treatment, monitoring, or improving care. The interviewers then focused the respondents to formulate their ideas into appropriate CER phrasing. A list of CER topics was produced by each of the two focus groups.

Prioritization of Proposed CER Topics and Analysis

All candidate CER studies were compiled into 21 topic categories (Supporting Table 2). Duplicate CER topics were combined and proposed topics that did not meet key elements of the IOM definition of CER were excluded. A two-person team (A.S.C. and C.A.S.) worked together to bin and winnow CER topics within each of our set of 21 CER domains.5 The two reviewers evaluated each CER topic generated from BRIDGe members, surveyed physicians and patients, and determined how best to categorize each CER within our conceptual framework of CER priorities. We established key words for each bin and assigned each CER topic a bin label. We then deleted CER topics (winnowed) from the CER library for 1) lack of face validity, or 2) redundancy with other CER topics. The purpose of winnowing was to eliminate unworkable CER topics rather than to arbitrarily select the best CER topics. CER topics were rejected due to face validity concerns for reasons including inconsistency with the IOM definition, vaguely written CER topics, and confusing CER topics. Following the review of all items, we updated the CER library to reflect whether the CER topic was selected for subsequent review in our RAND panel.

The candidate CER studies were then presented to the BRIDGe group using the validated RAND/UCLA methodology.6 This approach employs a modified Delphi panel to confidentially and iteratively rate the priority of the proposed CER studies using a standard 1–9 point scale (1 = lowest priority; 9 = highest; ≥7 = high priority). The first round of rating was performed individually and confidentially with deidentified responses via the Internet.

Following the initial ratings the panel convened at a moderated, in-person, meeting to anonymously re-rate priorities after detailed group discussion of the proposed CER topics. After the second round of rating the RAND/UCLA disagreement index was calculated to quantify the level of disagreement.6 A disagreement index greater than 1.0 indicates extreme variation, whereas disagreement index values less than 1.0 reflect increasing agreement.

As a final step to better define the 10 most important CER topics, each BRIDGe member was then given 55 “points” to allocate to any of the 15 highest-priority CERs, with up to 10 points allocated to any single CER. The total points of 55 was chosen to allow for up to 10 points to be given to a single CER and sequentially lower points to less important CER topics to create a rank order from the first to tenth priority. However, the points could be distributed as each BRIDGe member saw fit, with up to a maximum of 10 points per topic. This final voting process mirrored that of the IOM where members were given 300 points and could allocate up to 10% to any single topic. This final stage was carried out after the in-person meeting via the Internet.

The Institutional Review Board at Dartmouth-Hitchcock Medical Center issued a waiver of required review for this study.

RESULTS

Table 1 displays the characteristics of the survey respondents. In all, 245 respondents returned their surveys (6.3% response rate). Approximately half of the respondents were from the United States (51%) and 66% had been in practice for at least 10 years. Eighty-two percent of the physicians were male. Over 50% of the respondents saw, on average, more than 20 IBD patients a month and 60% estimated that 25% or less of their practice was IBD. Forty-three percent of physicians described themselves as being in private practice.

Table 1. Demographics of Complete Respondents (N = 139) Location of Practice N (%) Canada 10 (7.2%) U.K. 29 (20.9%) Australia 14 (10.1%) U.S. – Northeast 30 (21.6%) U.S. – Southeast 13 (9.4%) U.S. – Midwest 21 (15.1%) U.S. – Southwest 5 (3.6%) U.S. – West 14 (10.1%) Other 3 (2.2%) Years in practice  < 5 years 31 (22.3%)  5–10 years 17 (12.2%)  10–20 years 40 (28.8%)  > 20 years 51 (36.7%) Age  < 35 13 (9.4%)  36–45 56 (40.3%)  46–55 30 (21.6%)  56–65 31 (22.3%)  > 65 9 (6.5%) Gender  Male 114 (82.0%)  Female 25 (18.0%) Number of IBD patients seen in a 4-week period  None 2 (1.4%)  1–5 13 (9.4%)  6–10 19 (13.7%)  11–20 31 (22.3%)  > 20 74 (53.2%) Percentage of patients with IBD  < 10% 40 (28.8%)  11–25% 43 (30.9%)  26–50% 18 (12.9%)  > 50% 38 (27.3%) Practice setting  Private individual practice 12 (8.6%)  Private group practice 48 (34.5%)  University practice 38 (27.3%)  Managed care practice 7 (5.0%)  VA Hospital practice 11 (7.9%)  Academic: Clinical research 22 (15.8%)  Academic: Basic science research 1 (0.7%)

Respondents generated 341 candidate CER topics: 111 from the sample of U.S. gastroenterologists, 70 from the group of international gastroenterologists, 140 from the BRIDGe group, and 20 from the patient focus groups. The justification for listing a particular CER as a priority, the research areas of interest, and the specific patient populations proposed by the gastroenterologists are reported in Tables 2, 3, and 4.

Table 2. Justification for Listing a Proposed CER as a Priority Reason for Priority N (%) Variability in care 110 (10.1%) Cost 130 (11.9%) Information gap 162 (14.9%) Controversy 107 (9.8%) Public interest 48 (4.4%) Disproportionate impact by subpopulation 26 (2.4%) Answer will likely influence practice 251 (23.0%) Answer important for clinical decision making 235 (21.6%) Other 21 (1.9%) *N (Respondents) = 318. *N (Responses) = 1090. Table 3. Research Area Proposed CER Is to Address Area CER Will Study N (%) Biologics 48 (17.1%) Immunomodulators 34 (12.2%) 5-ASA therapy 10 (3.6%) Antibiotics 5 (1.8%) CAM therapy 4 (1.4%) Surgery 7 (2.5%) Risks of therapy 23 (8.3%) Postoperative management 9 (3.2%) Pharmacogenomics 2 (0.7%) Biomarkers 7 (2.5%) Imaging modalities 10 (3.6%) Preventing colon cancer 18 (6.5%) Pregnancy 1 (0.4%) Pediatrics 4 (1.4%) Extraintestinal manifestations of IBD 2 (0.7%) Etiology/risk factors of IBD 6 (2.2%) Osteoporosis 3 (1.1%) Shared decision-making 7 (2.5%) Patient reported outcomes 16 (5.8%) Patient decision support tools 11 (4.0%) Cost-effectiveness 40 (14.4%) Other 11 (4.0%) *N (Respondents) = 139. *N (Responses) = 278. Table 4. Study Populations for Proposed CER Topics Study Population N (%) Crohn's disease 224 (18.6%) Ulcerative colitis 166 (13.8%) Adults 210 (17.5%) Children/adolescents 145 (12.1%) Women 146 (12.1%) Men 143 (11.9%) Elderly 67 (5.6%) Ethnic subpopulation 39 (3.2%) Pregnant women 29 (2.4%) Pouch patients 23 (1.9%) Other 10 (0.8%) *N (Respondents) = 323. *N (Responses) = 1202.

By combining duplicate answers (n = 98) and eliminating proposals that could not be considered CER (n = 9), the 341 responses were collated into 234 unique CER questions (Fig. 1). In the first round, each of these 234 CER studies were rated for priority by each BRIDGe member on a scale of 1–9 using the RAND method described above. Based on this initial round of ranking, group discussion, and clinical relevance, the BRIDGe group (RAND panel) selected 87 CER questions to re-rate, of which 40 were rated as high priority (mean score of 7 or greater). Disagreement, based on the RAND/UCLA criteria,6 was observed in only 5 of 87 CER questions. Finally, using the point allocation system, the 15 highest priority CER topics (scores >7.5) were distilled to the final top 10 CER studies (Table 5; Supporting Table 3).

Study design. CER, comparative effectiveness research; ACG, American College of Gastroenterology; BRIDGe, Building Research in Inflammatory Bowel Disease Globally; IBD, inflammatory bowel disease.

Table 5. Top 10 Comparative Effectiveness Research Studies in IBD Final Rank Compare the Effectiveness of… Points Allocated in Final Round of Ranking Previous Rank from Top 15 List 1 Using biomarkers (genetics, immunology) versus standard of care in predicting response and toxicity to therapy of IBD. 83 2 2 Withdrawal of anti-TNF agent versus withdrawal of immunomodulator therapy versus continued combination therapy in Crohn's disease patients who achieved remission on combination therapy. 80 1 3 Tailoring therapy to mucosal healing versus standard clinical factors on long-term outcomes in IBD. 71 9 3 Using biomarkers versus standard clinical data to predict disease prognosis in new onset IBD. 71 15 5 Standard anti-TNF agent dosing versus trough-level anti-TNF agent dosing in IBD. 61 5 6 Anti-TNF agent alone versus anti-TNF agent + thiopurine in patients with moderate to severe Crohn's disease failing thiopurine. 54 8 7 Efficacy and safety of long term combination anti-TNF agent + immunomodulator therapy beyond 1 year versus mono-therapy with anti-TNF agent versus monotherapy with immunomodulator 52 11 8 Thiopurine versus anti-TNF agent in preventing clinical and endoscopic post-op recurrence of Crohn's disease after resection. 37 3 8 Efficacy and safety of infliximab versus cyclosporine versus surgery in hospitalized steroid- refractory ulcerative colitis. 37 6 8 Efficacy and safety of long-term immunomodulation versus anti-TNF therapy in Crohn's disease and UC 37 10

None of the CER topics proposed by the patient focus groups made the top 10 and the only CER study proposed by the patients in the top 15 was comparing the effectiveness of competing dietary interventions.

The most frequent responses for why the CER topics should be a national priority were that the answers would likely influence practice, they were important for clinical decision-making, and that there was a knowledge gap between what is known and what clinicians want to know to guide decisions. The most common areas selected for study were biologics (17.3%), immunomodulators (12.2%), and cost-effectiveness (14.4%).

DISCUSSION

We used a systematic approach mirroring the methods applied by the U.S. IOM to develop a priority list of CER studies for IBD. Two of the top 10 CER topics (including the top-rated CER study) focused on the use of biomarkers versus standard of care to predict prognosis, outcomes, and response to therapy. Optimizing safety and efficacy of biologics in IBD was exceedingly important to stakeholders and was the primary focus of 5 of the top 10 CER. Not surprisingly, respondents believed that the CER topics they submitted would likely influence practice, were important for clinical decision-making, and would help to close an information gap between what is known and what clinicians would like to know to guide clinical practice.

CER is not a new concept in IBD. There are many examples of excellent CER studies in the field of IBD; these share the goal of providing practical answers to questions that affect the day-to-day care of patients. One of the best examples is the recently published SONIC trial, which compared the use of azathioprine versus infliximab versus the combination of the two drugs in patients with moderate-severe Crohn's disease who were naïve to immunomodulators and biologics.7 Another recent CER study in IBD is the so-called “top-down, step-up” trial in Crohn's disease, which compared the competing treatment strategies of early aggressive therapy with combination infliximab and azathioprine induction with azathioprine maintenance to “conventional” therapy with multiple courses of corticosteroids followed by the sequential addition of azathioprine and then infliximab only if necessary.8 However, according to a recent review, most studies performed in the field of IBD do not meet CER criteria.9

Our study has limitations. First, only 8% surveyed U.S. gastroenterologists returned our survey. The response rate (5.4%) was even lower from international physicians, likely due to the fact that the request to participate in the study was part of a larger email bulletin and they had less time to respond. However, our response rate was similar to the experience of the Institute of Medicine study, which surveyed over 20,000 individuals and received 1758 responses (less than 8.8%) and 1268 unique CER topics.1 Additionally, it is important to distinguish this survey from a traditional survey that aims to draw generalizable conclusions about knowledge, attitudes, beliefs, or practice patterns. In this instance, we cast a wide net to draw in ideas for comparative effectiveness research, not to draw representative conclusions about community practice patterns. The survey was also subject to a responder bias, as the respondents likely represented individuals with a vested interest and expert knowledge in IBD, including the BRIDGe group. In this survey 27% of responders have practices with >50% IBD, which is considerably higher than a general community practice. Another limitation is that we did not specifically ask gastroenterologists or patients to take cost or the feasibility of the study into consideration. Although the practicalities and cost of the studies are a critical consideration if they are eventually undertaken, our main goal was to prioritize the most important, clinically relevant topics that require further research irrespective of cost or feasibility.

To our knowledge, this is the first attempt by a group, other than the Institute of Medicine, to identify the most important CER topics within their specialty field. We demonstrated that this process is feasible outside of a government-sponsored setting. We acknowledge that the prioritized list of top 10 CER topics is clearly the opinion of those who participated in the process and there are many other important day-to-day clinical questions that are relevant to the care of persons with IBD. Yet we propose that this systematic process yielded a list of CER topics that may be useful for researchers and funding agencies as a “roadmap” for prioritizing future directions of research. Since the IBD knowledge base is constantly changing, like the IOM, we propose that identifying and prioritizing IBD CER topics should be an ongoing process of continuous reevaluation and updating as conditions change and the impact of previous CER become evident. The methodology we used is feasible and could also be repeated as part of an ongoing reevaluation process and utilized by other groups interested in developing a priority list of CER in their particular specialties.

CONCLUSIONS

We systematically developed a list of high-priority CER topics for IBD based on a survey of gastroenterologists, expert review, and patient input that may be useful for guiding IBD research priorities that would impact day-to-day clinical practice. We demonstrated feasibility of the process outside of a government-sponsored setting, which may be helpful for other subspecialty groups interested in developing a list of leading CER topics for their specialties.

Acknowledgements

The authors thank Dr. Harold Sox for contributions in consulting on study design and following IOM methodology and for article review. Disclosures: A.S.C.: Advisory Board (Warner Chilcott, Abbott, UCB); G.Y.M.: Consultant and/or Research Support (Centocor, Setpoint, Amgen, Celgene), Speaker Bureau (Abbott, Prometheus), Advisory Board (UCB); C.A.S.: Advisory Board (Abbott, UCB, P&G), Consultant and Research Support (Elan, P&G), CME activities (Abbott, UCB); B.M.S.: Research (Shire Pharmaceuticals, Ironwood Pharmaceuticals, Takeda Pharma, Rose Pharma A/G), Advisory Board (Prometheus, Salix, Boehringer Inngelheim, Ironwood, AstraZeneca, Pfizer); B.B.: Advisory Board (Abbott, Schering-Plough Canada, Shire, Ferring, Merck, Warner Chilcott, Genentech), Speaker Bureau (Janssen Ortho, Abbott, Merck, Shire, Takeda, Ferring, Warner Chilcott) Research Support (Abbott, Merck, Amgen, GlaxoSmithKline, Millenium Pharmaceuticals, Genentech, Bristol-Myers Squibb); J.J.: Consulting, Speaker's Bureau and Educational Funding (Schering-Plough Canada, UCB Pharma, Shire, P&G and Abbott); M.S.: Research Support (Ferring Pharmaceuticals, Abbott Pharmaceuticals), Advisory Board (Schering-Plough Pharmaceuticals); F.V.: Research Support (P&G); P.M.I.: Advisory Board (Abbott, MSD, Shire, Falk), Speaker's Bureau (Shire, Abbott, MSD, Merck); S.M.D.: Speaker (Schering-Plough, Abbott, Merck, Shire), Advisory Board (Abbott, UCB, Schering-Plough, Shire, Merck); G.G.K.: Advisory Board (Schering-Plough, Abbott, UCB, Shire); L.H.: Consultant (Abbott, Salix), Research support (Glycominds); L.B.: Advisory Board (UCB), Speaker's Bureau (Centocor, Salix); P.K.: None. Contributions: A.S. Cheifetz: study concept and design, drafting of the article, acquisition of data, analysis and interpretation of data, critical revision of the article for important intellectual content of the article; G.Y. Melmed, B. Spiegel, C.A. Siegel: study concept and design, acquisition of data, analysis and interpretation of data, critical revision of the article for important intellectual content of the article; J. Talley: acquisition of data, analysis and interpretation of data, critical revision of the article for important intellectual content of the article; S.M. Devlin, L. Harrell, P.M. Irving, J. Jones, G.G. Kaplan, P. Kozuch, M. Sparrow, F. Velayos, L. Baidoo, B. Bressler: acquisition of data, interpretation of data, critical revision of the article for important intellectual content.

Additional Supporting Information may be found in the online version of this article.

Filename Description IBD_22920_sm_SuppTab1.doc22.5 KB Supporting Table 1. BRIDGe Group: Comparative Effectiveness Research Priorities in Inflammatory Bowel Disease (IBD) Survey IBD_22920_sm_SuppTab2.doc19 KB Supporting Table 2. Areas of CER research IBD_22920_sm_SuppTab3.doc22.5 KB Supporting Table 3. Top 15 Comparative Effectiveness Research Studies in IBD prior to final point allocation

Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.

REFERENCES

1 Ratner R, Eden J, Wolman D, Greenfield S, Sox H, eds. Initial national priorities for comparative effectiveness research. In: Io Medicine, ed. Washington DC: National Academies Press; 2009. 2 Sox HC, Greenfield S. Comparative effectiveness research: a report from the Institute of Medicine. Ann Intern Med. 2009; 151: 203– 205. 3 Siegel CA. Accidentally ASCENDing into comparative effective research for inflammatory bowel disease. Gastroenterology. 2009; 137: 1880– 1882. 4 Ericsson KaSH. Protocol Analysis: Verbal Reports as Data (Revised Edition). Cambridge, MA: MIT Press, 1993. 5 DeWalt DA, Rothrock N, Yount S, et al. Evaluation of item candidates: the PROMIS qualitative item review. Med Care. 2007; 45: S12– 21. 6 Brook R. The RAND/UCLA Appropriateness Method. Clinical Practice Guideline Development: Methodology Perspectives. Rockville, MD: AHCR; 1994. 7 Colombel JF, Sandborn WJ, Reinisch W, et al. Infliximab, azathioprine, or combination therapy for Crohn's disease. N Engl J Med. 2010; 362: 1383– 1395. 8 D'Haens G, Baert F, van Assche G, et al. Early combined immunosuppression or conventional management in patients with newly diagnosed Crohn's disease: an open randomised trial. Lancet. 2008; 371: 660– 887. 9 Slattery E, Harewood GC. Comparative effectiveness research in inflammatory bowel disease. Inflamm Bowel Dis. 2011; 17: E46– 47.