Consensus methods are used in medical education and health services research to synthesize opinions and prioritize responses for decision-making, education, and assessment, among others.1,2 In the context of primary care, techniques such as the nominal group technique (NGT), which aims to extract opinions and consensus on a nominal question, have been relied on heavily to synthesize expert opinions to develop consensus care guidelines.2,3 For example, the NGT has been used to evaluate exercise adherence measures for patients with musculoskeletal disorders.4 In continuing professional development (CPD), NGT has been used for small group learning and needs assessment.5–7 However, there is a paucity of evidence and standardization in using consensus methods for generating patient-reported experience measures' (PREM) items for quality improvement and medical education more broadly.8,9 For more effective NGT implementation in primary care and CPD, there should be an emphasis on composing a more diverse panel, beyond physician experts alone who compose existing CPD programming and care guidelines, to encompass varied patient–physician interactions and adapt training to trends in healthcare delivery goals.10,11

One such trend is the emphasis on prevention as seen in cardiovascular care. Understanding patients' experiences living with a health condition and their preferences for primary care services represent important parts of evaluating and improving care delivery to improve patient outcomes. These preferences should be incorporated into existing CPD programming and care guidelines.11 Thus, using cardiovascular disease (CVD) risk factor reduction in primary care as an example, we describe a methodological study on the use of the NGT by both patients and physicians to codetermine important PREM items. In doing so, we hope to aid in the development of effective assessment tools that capture patient experience, guide CPD, evaluate practice interventions, and ultimately improve practice.9,12 Moreover, readers can follow the steps in our example for consideration in their own projects.

METHODS Study Design

This study described the involvement of both patients and physicians using NGT to build a consensus on cardiovascular health outcomes to be used as patient-negotiated assessment items for primary care and CPD. NGT was selected as it is a well-validated method for reaching consensus, especially based on input from experts.1,13 In brief, NGT involves structured and iterative discussions between 5 and 12 participants with the goal of bringing individually and privately generated ideas to consensus among the larger group. Through this process of discussion, a large number of heterogenous inputs are created, each with the possibility of combination with others. The small number of 5–12 participants allows for robust and dynamic discussion during idea generation and is also more manageable for the facilitator should participants need facilitation or assistance.14 Compared with other consensus methods such as Delphi, it is more commonly used among lay people and its facilitated nature ensures that all members are involved in idea generation and voting.9 We followed a recommended and reproducible consensus group method protocol.15

Participation/Recruitment

Ethics approval was obtained at the local Research Ethics Board at the Montfort Hospital and Bruyère Hospital. Five patient members of the Montfort Hospital patient partners program with CVD treatment experience were invited to participate in the study. Notices for recruitment of physician panel members were sent to eight total eligible physician participants within the Ottawa Hospital and Montfort Hospital, two large academic teaching hospitals affiliated with the University of Ottawa. As part of the Canadian Institute of Health Research Strategy for Patient-Oriented Research initiative, which is a national initiative that encourages research to focus on patient-identified priorities, we recruited four patient partners from the Montfort Hospital to participate in the nominal group session. Four primary care providers (PCPs) from across the Ottawa region also participated in the session. These physicians were recruited through connections of team members seeking a purposive sample with diverse ages, gender, and practice models. All participants provided informed written consent to participate in the research study.

Procedure, Data Collection, and Analysis

A skilled moderator and expert in NGT led the session in June 2019, which was conducted in English. The project principal investigator first introduced the group to the objectives and purpose of the project, as well as the Canadian Primary Care Information Network (CPIN). This is an automated patient communication and survey system through which the PREM items developed by the group would be trialed as short patient surveys delivered by email or text message. Furthermore, the group was presented with a sample list of measures used to evaluate CVD preventative care and shown examples of high-quality patient education materials. This served to familiarize participants with current literature and help guide their ideas.

Initial items generation: All participants were asked to privately write down any items that they felt would be important in evaluating the quality of CVD preventative care delivered in primary care practices. Each participant was given 10 minutes to write down items as either questions or topics without restrictions to the content or form, in accordance with established protocols from other studies.16,17 A total of 26 items (Table 1) were recorded on the flipchart paper by an investigator. Discussion and combination of items: Subsequent group discussions to combine similar items brought the total number to 22before private ranking. Rounds: In a group setting, each member of the panel was asked to share their own ideas in a round robin format and moderated discussion subsequently took place for each item. Emphasis was placed on achieving collective understanding of each item. New ideas were generated, irrelevant items were excluded, and similar items were combined. This took place until all items from each panel member were discussed, and no new ideas were forthcoming. Throughout this process, the moderator ensured that there was equal participation among panel members but limited direction to avoid interference with group process; no individual dominated a discussion. TABLE 1. - Codetermination of PREM Items During NGT Item # Initial items generated (n=26) Discussion and combination of items (n=22) Round 1 (n=9) Round 2 (n=4) Combination of items (n=1) Top 10 ranked (n=10) 16 Is your treatment plan tailored to you? I (Pt) 1 7 How good was I at giving information about risk factors? I (P) 2 23 Did I involve you in decision-making? I (P) 3 12 How comfortable are you talking about your risk factors? I (Pt) 4 13 To what extent do you feel your physician has your best interest in mind? I (Pt) 5 10 Were all your questions about CVD medications answered? I (Pt) 6 15 Did your physician give you good advice for exercise and healthy habits? I (P) 7 24 Did I explain your risk level well enough? I (Pt) T-8a 25 Do you know how to get help in case of a CVD emergency? I (Pt) T-8b 20 Did I advise you on ways to reduce your blood pressure? I (P) 10 1 Is CVD inherited? E 2 Do you know your blood pressure in the last year? C E 3 Did I inform you about your blood pressure results? 4 Are there factors besides blood pressure that contribute to heart attack/stroke risk? E 5 Do you know the signs/symptoms of heart disease/stroke? E 6 Do you want more information about risk factors? E 8 Would it help if I provided you with websites for more info? E 9 Could you tell your friends/family about your CVD situation? E 11 Were you given a follow-up for new medication? I R E 14 Do you need more info about community resources such as exercise facilities, smoking cessation, and support groups? E 17 Did you have your blood pressure measure in the last year? C I R E 18 How many visits did you have your blood pressure measured? 19 Did I ask you to monitor your blood pressure and report back if elevated? E 21 Did I explain CVD properly? E 21 Did I explain the medications properly (side effects)? I R E 26 Did I demonstrate awareness of continuity of care (up to date)? E

C, combined; CVD, cardiovascular disease; E, excluded; I, included; NGT, nominal group technique; P, primary care physician-centered; Pt, patient-centered; R, recombined; T, tie.

All panel members were then asked to anonymously evaluate each of the items generated based on their importance and relevance in being included in a patient questionnaire evaluating the quality of CVD preventative care delivered in primary care practices. Items were ranked on a five-point scale, with five being highest priority and one being lowest priority.18 It was determined a priori that questions receiving 75% agreement of ranking 4 to 5 were included, whereas those receiving 75% agreement of ranking 1 to 3 were excluded. Scores were then tallied by the moderator. Any items not achieving consensus were subject to subsequent group discussion and further rounds of reranking. A total of two rounds of reranking were performed, as typically seen in NGT, to produce the top 10 items. Ten items was considered a priori as a reasonable number for patients to complete on a survey. The entire process took about 3 hours to complete.

RESULTS

A total of eight participants joined in the in-person NGT session. Among patient partners (n = 4), all represented both English-speaking and French-speaking populations within the Ottawa area. Among PCPs (n = 4), two joined remotely through telephone, and there were two servicing three socially diverse neighborhoods in the community.

Figure 1 outlines the NGT indicator generation process. Items that were included and excluded during successive rounds of discussion and ranking are listed in Table 1. In brief, 26 items were generated from initial independent brainstorming by panel members. After discussion to eliminate or combine similar items, 22 possible indicators entered the first round of rankings.

FIGURE 1.:

Item generation process during NGT. NGT, nominal group technique.

Two iterations of group discussions and independent ranking were performed to identify the top 10 PREM items (Table 1, thick border) for assessing the quality of CVD prevention. After the first round of ranking, nine items met a priori criteria for inclusion. No items achieved a priori criteria for exclusion. Thirteen remaining items continued to a second round of discussions and independent ranking. Of these, four items met criteria for inclusion, whereas nine items met criteria for exclusion. After the second round of discussions, the final list of items was revisited to combine similar items based on discussion and consensus among the group members. Two items were combined, resulting in 10 total items generated. Four of the top 10 items were physician-centered, and 6 items were patient-centered assessment items.

DISCUSSION

Here, 10 highly ranked, patient and physician codetermined PREM items for evaluating primary care of CVD prevention are identified. Central themes of communication, patient–physician relationship, shared decision-making, and communication of risk and risk reduction methods underlie shared interests between patients and physicians. By bringing together physicians and patients with lived experience, we replicated the crosstalk, dynamics, and exchanges that occur during clinical encounters that are critical to influencing outcomes. This study suggests that measuring quality improvement or performance with typical indicators or disease targets alone may not completely capture the goals of care. Patients and physicians may have different views on specific factors that contribute to high-quality CVD care, but they may work together to reach a consensus.19 This allows for the most accurate representation of the diverse views, values, and preferences of panel members while expanding on existing quality indicators that focus on evidence-based processes, disease management, or prevention targets.11,19,20 In doing so, this study identifies aspects of cardiovascular care that may not be identified by physicians alone in generating existing quality indicators.19

Our results emphasize that, when considering patient perspectives in addition to physician priorities in the evaluation of cardiovascular primary care quality, items emphasizing the therapeutic relationship are prioritized over more conventional measures currently used. These are in line with the Quadruple Aim for Health Systems, which emphasizes improving patient experience as a key to delivering effective health care.21 These results also mirror the priorities for general primary care performance measurement and reporting identified by in-depth engagement with patients, as well as work conducted in priorities of dialysis outcomes.22–24 Only item 7 and 10 allude to traditional modifiable risk factors described in major Canadian, American, and United Kingdom guidelines.25–27 Items one and three are a direct reflection of the long recognized importance of shared decision-making in the context of management of CVD and medication initiation.19,28 Although the 2019 American College of Cardiology/American Heart Association (ACC/AHA) Guidelines recommend that health care providers engage patients in discussions about cardiovascular risk estimates and their implications in perceived benefits of preventative care, there is significantly less clarity about how to structure these types of discussions. This study suggests that risk factor communication may be a defining characteristic of high-quality cardiovascular primary care; it is a meaningful and clinically useful common ground between patients and physicians when engaging in shared decision-making (items 2, 4, 8a, and 10). For patients, their ability to understand risk depends on factors including their perceived level of control over their own health (item 4) as well as the concordance in communication styles between patients and physicians (items 2 and 8a). For physicians, risk communication involves first knowing the guidelines and novel research applicable to the clinical situation, understanding how this evidence applies to the individual patient, and presenting the risks in a way that is easy to understand (items 2 and 8a). Evidence shows that patients generally have a poor understanding and appreciation of CVD risk, and the way in which risk is communicated can influence patient comprehension, patient satisfaction, and may encourage adherence.29 Strategies for improving risk communication have been reviewed elsewhere.30

Our findings contrast most available CVD outcome measures both inside and outside the primary care setting, which are developed by physician panels and medical expert discussion alone.31–37 Studies have shown that physicians often mistakenly believe they know what patients' preference are about the initiation of medications for prevention of CVD, resulting in suboptimal shared decision-making.19 They concluded that patient values are often unpredictable and unlikely to be captured by guidelines as they are currently developed. Similar results have been demonstrated among patients starting antihypertensive medications.28 Furthermore, both the CANHEART Performance Indicators for Cardiovascular Ambulatory Care and ACC/AHA indicators use clinical outcomes as the primary measures for quality of care delivered,35–37 whereas it is clear that patient experience does not correlate well with hard outcomes. If improving CVD care requires attention to and even measuring of patients' experience and trust in their health care provider, additional approaches to evaluating care to ensure routine engagement of patients in assessing their experience of care and communication may be required alongside typical chart audit and administrative data analyses.38

Further refinement (through criterion referenced validity and member checking) and pilot testing of the codesigned PREM items will be taking place in four Ottawa clinics using an automated patient survey system (CPIN). The reporting of these results may contribute to CPD by enabling comparisons in patient experience, sharing of patient ratings of outreach educational material, and sharing of the different types of information and surveys peers are using. As CPIN expands, this may lead to sharing information across borders. The creation of targets based on survey responses may be useful in improving individual practices by giving individual physicians feedback and comparisons on the subjective experiences of patients they are treating. This would build on objective feedback that physicians receive regarding their care, as delivered through patient report cards.12 Further efforts to design measurement approaches which incorporate shared priorities, or acknowledge distinct priorities in performance goals, may offer a better understanding of our health system performance and more broadly.

Strengths and Limitations

Our process for identifying priority PREM items is unique in that patients, and providers were equally represented, and group facilitation ensured equal participation during the NGT process. In addition, no restrictions were placed on whether measures involved access, process, or continuity of care, outcomes, or patient–physician interaction. This is in contrast to the typical method used for developing performance measures for clinical care.31–37 There are a number of limitations to this study example. First, it is a single center study involving participants from Ottawa, Canada, and CPIN is for now a regional and national level program. Therefore, it represents the views, priorities, and preferences of a group operating under a universal, single payer, and publicly funded health care system. The culture of patient and physician dialog may also vary between health care systems based on time, funding, and societal factors. As well, the selection of patient partners willing to participate in the NGT may skew results toward those most willing to be involved in their own care. This underlies the importance of patient–physician communication in all types of encounters to establish expectations and individual preferences.39 Finally, although the involvement of eight total participants optimizes communication between panel members, it inevitably reduces the number of opinions that are considered. Expert moderation of NGT was used to ensure that all opinions, where present, were heard and discussed.

CONCLUSIONS

Health services' research is increasingly valuing the perspectives of patients when evaluating performance through the use of patient-reported outcomes, which can be identified using NGT.11,21 In this example, we have generated PREMs which include highly ranked items on CVD prevention, which may be valuable in guiding PCPs in their practice improvement efforts. These indicators are different compared with existing measures as they are the product of interactions between patients and physicians, thereby presenting a unique perspective on patient experience that is not adequately captured by existing quality indicators. Further work to explore the feasibility, utility, and impact of these and similar PREM items will advance the field of performance measurement in health care.Lessons for Practice ▪ The NGT can be used as a strategy in involving patients in the CPD planning process. ▪ Patient-negotiated assessment items generated through NGT can guide the evaluation of CPD programs. ▪ The patient as a member of the health care team should be empowered to be involved in the development of CPD programs.

ACKNOWLEDGMENTS

The authors thank Dr. Michael Fitzgerald, Ph.D., who provided research assistant support for this study.

REFERENCES 1. Humphrey-Murto S, Varpio L, Wood TJ, et al. The use of the Delphi and other consensus group methods in medical education research: a review. Acad Med. 2017;92:1491–1498. 2. Søndergaard E, Ertmann RK, Reventlow S, et al. Using a modified nominal group technique to develop general practice. BMC Fam Pract. 2018;19:117. 3. Campbell S, Braspenning J, Hutchinson A, et al. Research methods used in developing and applying quality indicators in primary care. Qual Saf Health Care. 2002;11:358–364. 4. Mallett R, McLean S, Holden MA, et al. Use of the nominal group technique to identify UK stakeholder views of the measures and domains used in the assessment of therapeutic exercise adherence for patients with musculoskeletal disorders. BMJ Open. 2020;10:e031591. 5. Kennedy A, Clinton C. Identifying the professional development needs of early career teachers in Scotland using nominal group technique. Teach Dev. 2009;13:29–41. 6. De Villiers M, Bresick G, Mash B. The value of small group learning: an evaluation of an innovative CPD programme for primary care medical practitioners. Med Educ. 2003;37:815–821. 7. Moore DA, Klingborg DJ. Involving practitioners in continuing education needs assessment: use of the nominal group technique. J Vet Med Educ. 2007;34:122–126. 8. Humphrey-Murto S, Varpio L, Gonsalves C, et al. Using consensus group methods such as Delphi and Nominal Group in medical education research. Med Teach. 2017;39:14–19. 9. McMillan SS, King M, Tully MP. How to use the nominal group and Delphi techniques. Int J Clin Pharm. 2016;38:655–662. 10. Kulczycki A, Shewchuk RM. Using Internet-based nominal group technique meetings to identify provider strategies for increasing diaphragm use. BMJ Sex Reprod Health. 2008;34:227–231. 11. Krahn M, Naglie G. The next step in guideline development: incorporating patient preferences. JAMA. 2008;300:436–438. 12. Ferreira-Legere LE, Chu A, Rashid M, et al. Making Informed CHOICES: The Launch of a “Big Data” Pragmatic Trial to Improve Cholesterol Management and Prevent Heart Disease in Ontario. Healthc Q. 2020;22:6–9. 13. Jones J, Hunter D. Qualitative research: consensus methods for medical and health services research. BMJ. 1995;311:376–380. 14. McMillan SS, Kelly F, Sav A, et al. Using the nominal group technique: how to analyse across multiple groups. Health Serv Outcomes Res Methodol. 2014;14:92–108. 15. Tran C, Archibald D, Humphrey-Murto S, et al. What makes a high-quality electronic consultation (eConsult)? A nominal group study. J Telemed Telecare. 2020;26:239–247. 16. Miller D, Shewchuk R, Elliot TR, et al. Nominal group technique: a process for identifying diabetes self-care issues among patients and caregivers. Diabetes Educ. 2000;26:305–314. 17. Dening KH, Jones L, Sampson EL. Preferences for end-of-life care: a nominal group study of people with dementia and their family carers. Palliat Med. 2013;27:409–417. 18. Dewar A, White M, Posade ST, et al. Using nominal group technique to assess chronic pain, patients' perceived challenges and needs in a community health region. Health Expect. 2003;6:44–52. 19. Lewis DK, Robinson J, Wilkinson E. Factors involved in deciding to start preventive treatment: qualitative study of clinicians' and lay people's attitudes. Heart. 2004;90:180. 20. Hopkins J, Agarwal G, Dolovich L. Quality indicators for the prevention of cardiovascular disease in primary care. Can Fam Physician. 2010;56:e255–e262. 21. Bodenheimer T, Wagner EH, Grumbach K. Improving primary care for patients with chronic illness: the chronic care model, part 2. JAMA J Am Med Assoc. 2002;288:1909–1914. 22. Urquhart-Secord R, Craig JC, Hemmelgarn B, et al. Patient and caregiver priorities for outcomes in hemodialysis: an international nominal group technique study. Am J Kidney Dis. 2016;68:444–454. 23. Manera KE, Johnson DW, Craig JC, et al. Patient and caregiver priorities for outcomes in peritoneal dialysis: multinational nominal group technique study. Clin J Am Soc Nephrol. 2019;14:74–83. 24. Slater M, Abelson J, Wong ST, et al. Priority measures for publicly reporting primary care performance: results of public engagement through deliberative dialogues in 3 Canadian provinces. Health Expect. 2020;23:1213–1223. 25. Arnett DK, Blumenthal RS, Albert MA, et al. 2019 ACC/AHA Guideline on the Primary Prevention of Cardiovascular Disease: A Report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines. Circulation. 2019;140:e596–e646. 26. Tobe SW, Stone JA, Anderson T, et al. Canadian Cardiovascular Harmonized National Guidelines Endeavour (C-CHANGE) guideline for the prevention and management of cardiovascular disease in primary care: 2018 update. CMAJ. 2018;190:E1192–E1206. 27. National Institute for Health and Care Excellence. Cardiovascular disease: risk assessment and reduction, including lipid modification. London, United Kingdom: National Institute for Health and Care Excellence, 2016. Available at: https://www.nice.org.uk/guidance/cg181/. 28. McAlister FA, O'Connor AM, Wells G, et al. When should hypertension be treated? The different perspectives of Canadian family physicians and patients. CMAJ. 2000;163:403–408. 29. Webster R, Heeley E. Perceptions of risk: understanding cardiovascular disease. Risk Manag Healthc Policy. 2010;3:49. 30. Navar AM, Stone NJ, Martin SS. What to say and how to say it: effective communication for cardiovascular disease prevention. Curr Opin Cardiol. 2016;31:537. 31. Tu JV, Khalid L, Donovan LR, et al. Indicators of quality of care for patients with acute myocardial infarction. CMAJ. 2008;179:909–915. 32. Lopez-Sendon J, González-Juanatey JR, Pinto F, et al. Quality markers in cardiology. Main markers to measure quality of results (outcomes) and quality measures related to better results in clinical practice (performance metrics). INCARDIO (Indicadores de Calidad en Unidades Asistenciales del Área del Corazón): a SEC/SECTCV consensus position paper. Rev Esp Cardiol (Engl Ed). 2015;68:976–995. 33. Al-Khatib SM, Yancy CW, Solis P, et al. 2016 AHA/ACC clinical performance and quality measures for prevention of sudden cardiac death: a report of the American College of Cardiology/American Heart Association Task Force on Performance Measures. J Am Coll Cardiol. 2017;69:712–744. 34. Levine GN, Bates ER, Blankenship JC, et al. 2015 ACC/AHA/SCAI focused update on primary percutaneous coronary intervention for patients with ST-elevation myocardial infarction. J Am Coll Cardiol. 2016;67:1235–1250. 35. Burge FI, Bower K, Putnam W, et al. Quality indicators for cardiovascular primary care. Can J Cardiol. 2007;23:383–388. 36. Redberg RF, Benjamin EJ, Bittner V, et al. AHA/ACCF 2009 Performance Measures for Primary Prevention of Cardiovascular Disease in Adults: A Report of the American College of Cardiology Foundation/American Heart Association Task Force on Performance Measures (Writing Committee to Develop Performance Measures for Primary Prevention of Cardiovascular Disease): Developed in Collaboration With the American Academy of Family Physicians; American Association of Cardiovascular and Pulmonary Rehabilitation; and Preventive Cardiovascular Nurses Association. Circulation. 2009;120:1296–1336. 37. Tu JV, Maclagan LC, Ko DT, et al. The Cardiovascular Health in Ambulatory Care Research Team performance indicators for the primary prevention of cardiovascular disease: a modified Delphi panel study. CMAJ Open. 2017;5:E315–E321. 38. Wong ST, Johnston S, Burge F, et al. Measuring patient experiences: is it meaningful and actionable? Healthc Pap. 2017;17:22–29. 39. Hoffmann TC, Legare F, Simmons MB, et al. Shared decision making: what do clinicians need to know and why should they bother? Med J Aust. 2014;201:35–39.