STRENGTHS AND LIMITATIONS OF THIS STUDY

The level of agreement was above the preset required level of 80%.

Disagreement was largely determined by terminology of outcomes and misqualification of outcomes and not with the outcomes themselves.

We were unable to fully rule out selective participation in the consultation round.

Participants who agreed with the core outcome set after the face-to-face meeting could have been more likely to refrain from voting in the consultation round.

Introduction

Core outcome sets (COSs) have been developed to assist standardisation of trials, to reduce publication bias and harmonise reporting. A COS is an agreed minimum standard set of outcomes on a specific topic, prioritised by different stakeholders, including lay experts.1 The COS is considered to encompass the most relevant outcomes that should be reported in all trials, enabling comparison of studies and facilitating large data synthesis.

The COMET (Core Outcome Measures in Effectiveness Trials) Initiative has issued a number of documents to aid in the consensus methodology for developing a COS.2–4 The standard method is to start with a review of literature to identify all possible outcomes, then to converge to a consensus with all relevant stakeholders with a Delphi strategy. Finally, the prioritised list of outcomes is discussed in a face-to-face consensus meeting in which a consensus is reached on the final COS.

Although the methodology is well described, published evidence underpinning this methodology is limited. One item that lacks evidence is the procedure, timing, applicability and value of the consensus meeting.5–8

The consensus meeting has risks of bias. In contrast to the online Delphi procedure, the possibility of strong voice of the individual participant may affect voting behaviour of others in this non-anonymous meeting. Spoken and non-verbal communication can negatively influence the actual consensus and skew it towards individual opinions.

The COMET methodology advises that all stakeholder groups should have the opportunity to discuss the results, but it does not advise in the group size for the consensus meeting nor in the ideal distribution of different stakeholders.2 The consensus meeting is generally attended by a small portion of the total group, as well as an unequal representation of stakeholders, which results in selection bias. For example, in the study of van ’t Hooft et al, only 38 of 195 participants attended the consensus meeting, with an over-representation of obstetricians,9 and in the study of Egan et al, 14 of 151 participants attended the consensus meeting.10 Sometimes the consensus meeting is attended by participants who had not participated in the prior Delphi procedure. The timing of the consensus meeting at the very end of the total procedure gives the participants the possibility to make decisions with significant impact on the final COS outcome. It is assumed that the representatives of the stakeholder groups validly make a selection of the outcomes, without losing the strength of the much larger initial panel. This assumption has not been tested.

Our aim was to provide evidence if the decisions made by the consensus meeting on the final consensus COS were supported by the bigger online panel in the Core Outcome Set for the prevention and treatment of fetal GROwth restriction (COSGROVE) Study11 and the Definition and Core Outcomes on Hyperemesis Gravida (DCOHG) Study.12

Methods

All authors were involved in two recent COS development procedures that followed the COMET methodology.11 12 Both studies were registered prospectively with the COMET Initiative (registration number 689 and 805).

Consultation round

In both studies, a modified Delphi procedure was followed by a consensus meeting. After the consensus meeting, an additional round (the consultation round) was conducted to test if the decisions made in the consensus meeting were in line with the decisions of the bigger online panel. In this consultation round, all online participants of both procedures (COSGROVE and DCOHG), who had completed the Delphi round prior to the consensus meeting, were considered eligible and were approached to participate in the online consultation round. In this round, the final COS (as finalised during the consensus meeting) of the procedure they took part in was presented and participants were asked if they agreed or disagreed with the outcomes included in the COS. If the participant disagreed with the final COS, the participant would be asked to specify the inclusion or exclusion of outcomes they disagree to and to give argumentation for it.

Core Outcome Set for the prevention and treatment of fetal GROwth restriction: deVeloping Endpoints

The COSGROVE Study aimed to establish a COS for prevention and treatment studies of fetal growth restriction (FGR). Purposeful sampling was used to approach eight stakeholder groups: (1) users of maternity services (women and their partners) or their representative advocacy group; (2) midwives; (3) obstetricians; (4) paediatricians/neonatologists; (5) family doctors; (6) ultrasonographers; (7) policymakers; and (8) individuals with specific expertise/interest in research or perinatal care related to FGR. These groups were later combined into three groups: healthcare providers, researchers/academics and maternity service users. Potential participants were invited by email. The study consisted of three Delphi rounds followed by the consensus meeting in April 2018 in Brighton, UK in which all included outcomes were presented for discussion (figure 1A).

Figure 1

(A) Flow chart of participants in the COSGROVE Study. (B) Flow chart of participants in the DCOHG Study. COSGROVE, Core Outcome Set for the prevention and treatment of fetal GROwth restriction: deVeloping Endpoints; DCOHG, Definition and Core Outcomes on Hyperemesis Gravidarum.

The COSGROVE Study identified a single set of core outcomes that should be reported in trials evaluating interventions for both the prevention and treatment of FGR.

The consultation round was not part of the methodology of the original study and the outcomes were not used to change the developed COS. In the consultation round of the COSGROVE Study, participants who disagreed were asked to specify to which included or excluded outcome(s) they disagreed. In this procedure, the consultation round was only used as a tool to test the agreement of the online panel with the final COS after the consensus meeting.

Definition and Core Outcomes on Hyperemesis Gravidarum

The DCOHG Study aimed to develop a COS for studies evaluating interventions for treatment of hyperemesis gravidarum (HG). Sampling of participants with informed opinions and known expertise on HG was used and potential participants were invited by email.

The following stakeholder groups were identified: (1) researchers; (2) women with lived experience of HG and their families; (3) obstetric health professionals (obstetricians, gynaecologists, midwives) and (4) other health professionals involved in care for women with HG (general practitioners, dietitians, nurses).

The study consisted of two online Delphi rounds followed by the consensus meeting in October 2017 in Windsor, UK. Outcomes on which no consensus was reached in the online Delphi procedure were brought for discussion. Outcomes on which consensus (both on inclusion and exclusion) was reached during the Delphi procedure were only eligible for discussion if a participant had strong objectives against inclusion or exclusion in the COS of that specific outcome (figure 1B). In the consensus meeting, a preliminary COS was assembled.

In the DCOHG Study, the consultation round was part of the original study design and alterations could be made to the COS based on the outcomes of this round. In the consultation round of the DCOHG, study participants who disagreed to the COS were asked to motivate their disagreement in a free-text response. In this procedure, the participant comments following the consultation round were discussed by the steering group and, if deemed appropriate, were implemented in final COS formulation.

Patient and public involvement

COSGROVE: patients were included in the different Delphi rounds and in the face-to-face meeting to ensure that the outcomes that were included in the COS were relevant to them.

DCOHG: the patient perspective was included throughout all stages of the consensus project. Patients were included in the steering committee, in the different Delphi rounds, in the consensus development meeting and in drafting and revising the manuscript.

Statistical analysis

Descriptive statistics were used and data are presented in percentages and absolute numbers. Because of the decisive nature of the online consultation round, we chose to use a higher percentage as the predefined cut-off for agreement than usual in Delphi procedures.13 We predefined consensus to the COS based on the consensus meeting if at least 80% of agreement of the online panel was reached.

ResultsCore Outcome Set for the prevention and treatment of fetal GROwth restriction: deVeloping Endpoints

Of 238 registered participants, 180 (76%) finished the first round in the Delphi survey, of whom 59% (n=105) were healthcare providers, 29% (n=53) were researchers/academics and 12% (n=22) were maternity service users. One hundred seven (59%) completed the third round, of whom 16 (15%) participated in the consensus meeting (figure 1A). All stakeholder groups were represented in the consensus meeting (maternity service users (n=6), healthcare providers including midwives, obstetricians, neonatologists and family physicians (n=5) and researchers/academics in FGR (n=5)). Thirty-four prevention outcomes and 35 treatment outcomes were included based on the Delphi rounds and brought forward for discussion at the consensus meeting. After selection in the consensus meeting, the final COS included 22 outcomes in total. Given almost complete overlap, the consensus panel participants concluded that all outcomes were suitable for both prevention and treatment of FGR.

Eighty-three of 107 invited participants finished this consultation round (78%), 46% of initial participants. A total of 67 participants (81%) agreed with the final COS. The level of agreement in the individual parameters for either inclusion or exclusion ranged between 87% and 100% (table 1).

Table 1

Level of agreement in the consultation round for all included and excluded parameters in the consensus meeting, COSGROVE

One outcome (admission to high dependency (special care baby unit or neonatal intensive care unit)) was removed from the analysis, because it had been incorrectly presented as an included outcome in the consultation round. The outcome of the consultation round was, however, not changed by removing this item. The outcome was excluded in the consensus meeting because it was valued to be more subjective as measurement of neonatal well-being than the included outcome ‘need for mechanical ventilation’.

No single outcome was responsible for the disagreement of the 19% ‘disagreement’. On the included outcomes, there was a maximum of two experts who felt that eclampsia should have been excluded. Various excluded outcomes raised discussion, mainly because the respondent either did not read the explanation why some outcomes had been excluded or did not understand this explanation. For example: some outcomes were excluded in the consensus meeting because other (included) outcomes were considered to cover these outcomes; ‘intrapartum death’ was included in the definition of stillbirth and HELLP was covered by the included outcome of pre-eclampsia, following the revised ISSHP (International society for the study of hypertension in pregnancy) statement in 2014.14 Birth weight less than the 5th percentile was excluded, but birth weight below the 10th and the 3rd percentile was already included following the published consensus definition of growth restriction in the newborn.15 With other outcomes, there was just small (maximum 8%) disagreement with the results of the consensus meeting: the need for resuscitation and Apgar score were voted out in the consensus meeting because those were deemed unreliable outcomes, whereas the included need for mechanical ventilation was considered to cover a reliable reflection of the condition of the neonate shortly after birth.

Other outcomes that raised some disagreement for exclusion in the consultation round were considered to be important in the consensus meeting, but were considered to be baseline characteristics rather than outcomes. For example, the outcome abnormal fetal Doppler. Furthermore, Doppler assessment was considered as a ‘vague’ and unspecific term in the consensus meeting even as a baseline characteristic; it was agreed to stick to the well-defined published consensus definition of FGR that included Doppler abnormalities.16 Also, congenital and chromosomal anomalies were considered more applicable as baseline characteristics than outcomes in the consensus meeting.

Because of the agreement in general, the steering group decided that the raised discussion in the consultation round would not have been a reason for alternation of the COS (if the consultation round would have been part of the original study design).

Definition and Core Outcomes on Hyperemesis Gravidarum

Of the 277 invited participants, 178 (64%) finished the first round in the Delphi survey, of whom 39% of participants had lived experience (as a patient or carer) of HG. One hundred twenty-five (70%) completed the second online round (figure 1B). Of the 20 participants in the consensus meeting, 16 were selected from the online panel (this is 13% of all participants who completed the second online round), the additional 4 were invited only for the consensus meeting. All stakeholder groups were represented at the consensus meeting. Outcomes were presented for discussion at the consensus meeting; 26 included outcomes and 10 outcomes on which no consensus was reached after the second Delphi round. Five outcomes were merged and/or reframed into two outcomes. Of the 10 undecided outcomes, a total of 4 additional outcomes were included in the COS. This resulted in a total of 27 outcomes in the preliminary COS. Ninety-six of 125 invited participants finished the consultation study (77%, 54% of initial participants). A total of 81 participants (84%) agreed with the preliminary COS.

In the consultation round, 6 of 96 participants had the general comment that the COS consisted of too many outcomes. Overlap of two outcomes was pointed out in the consultation round. Based on the consultation round comments, the steering committee decided to merge the four outcomes into two more general descriptive outcomes. Four participants questioned if mental health was sufficiently covered by the outcome maternal mental well-being. Therefore, the steering group decided to change the wording of the outcome ‘well-being’ to ‘maternal physical and/or mental and/or emotional well-being’. In the consultation round, a total of five participants indicated a lack in causality of HG with the pregnancy complication outcomes included. After deliberation, the steering group decided that any core outcome in the COS could be independent of causality. One outcome was reworded because of comments in the consultation round. Two participants made suggestions to include validated score forms for outcomes. One participant disagreed with the exclusion of hypersalivation, and one to the exclusion of child’s neurodevelopment disorders, both had already been excluded by consensus in the Delphi procedure prior to the face-to-face consensus meeting. The final COS consisted of 24 outcomes (table 2).

Table 2

Included and reframed or rephrased outcomes after consensus meeting, DCOHG

Discussion

An online Delphi additional consultation round verified the opinion of participants in two COS Delphi consensus procedures and showed that the original online panel was in high agreement with the choices of the panel present at the face-to-face consensus meetings. This confirms that the face-to-face consensus meeting outcome reflects the opinion of the whole online panel.

In the COSGROVE Study, there was very little disagreement on the included outcomes; most disagreement was expressed in the list of outcomes that had been excluded at the consensus meeting, which means that these experts felt that the COS should have been expanded instead of shortened. A COS ideally represents a minimum of outcomes that should be reported. It does not restrict studies, as the excluded outcomes can be important for different study designs. The high percentage of agreement of the included outcomes is therefore vital.

The pattern of the discussion regarding disagreement to the final COS of the DCOHG was different. In this study, the outcomes that raised discussion in the consultation round were predominantly outcomes that were already included or excluded by consensus in the Delphi procedure prior to the consensus meeting, rather than in the consensus meeting itself. This could be caused by the fact that the consultation round is the first time that the panel sees the complete list of outcomes, instead of voting on individual outcomes. Also, the time between round 2 and the consultation round may help participants to freshly re-evaluate their previous answers. In the DCOHG Study, both consensus meeting arguments and consultation round arguments were considered by the steering group and led to small alterations of the COS.

At this moment, the COMET guidelines provide a minimum framework for the COS consensus procedure. In our opinion, general COS validity could benefit from stricter guidelines for the consensus meeting to reduce the current risks of bias. For example, to be able to correctly interpret the Delphi results, one should have participated in the Delphi. We advocate, therefore, to only invite participants of the Delphi procedure to the consensus meeting. Also, the consensus meeting should be limited to the items that have failed to reach a consensus in the Delphi procedure. Furthermore, all stakeholder groups that are participating in the Delphi procedure should be represented in the consensus meeting. To reduce the impact of the strong voice, there should be guidelines for adequate chairmanship of the meeting. But even though these changes will reduce risks of bias, they can not be fully eliminated.

A potential advantage of a consultation round is that any bias created by the selection of stakeholders, present at the face-to-face consensus meeting, is largely dissolved. Decisions made in the consensus meeting can be confirmed anonymously by the larger panel of participants, which adds face validity for implementation. The Delphi panel participating in the consultation round should not only be informed which decisions were made, but also how and why. Ideally, they receive a rapport of the consensus meeting along with the list of included and excluded outcomes in the final COS. Should the larger web panel not agree with the decisions from the consensus meeting, the consultation round may provide a safety net in preventing publication of a COS that lacks support in the field. If the consultation round does not reach the preset level of agreement, the outcomes with disagreement on their inclusion can be brought back until agreement is reached. Alternatively, the COS could be reduced strictly to the outcomes that reach the level of agreement. For this reason, agreement has to be determined separately for all outcomes and also to determine the nature of the disagreement: either true, or for example, because of not reading the instruction regarding merging of outcomes into one term (like HELLP and pre-eclampsia in the COSGROVE procedure). The consultation round can also expose additional argumentation to strengthen steering group decisions. This gives an option to further refine the final COS, even if agreement is high. None of these issues arose in the two COS procedures in which we piloted the consultation round, and we were unable to investigate the consultation round’s potential role in resolving disagreement, or enhancing COS validity or implementation. Future research could investigate these issues in larger numbers of COS procedures, which could help establish the position of the consultation round in COS development.

Strengths and limitations

In both studies, the level of agreement was above the preset required level of 80%. Moreover, disagreement was largely determined by terminology of outcomes and misqualification of outcomes (eg, baseline characteristic or outcome measure) and not with the outcomes themselves. The outcomes that the participants disagreed upon in the additional consultation rounds varied in both procedures, which means that the agreement on the individual outcomes was higher. Therefore, the presented percentages are conservative rather than opportunistic.

We were unable to fully rule out selective participation in the consultation round. Participants who agreed with the COS after the face-to-face meeting could have been more likely to refrain from voting in the consultation round, although this is not represented by a lower response rate comparing with the other Delphi rounds (figure 1A,B).

Conclusion

In conclusion, using a final consultation round, we found the COS Delphi procedure, including the face-to-face consensus meeting, to be a valid step within the COMET methodology in the development of COSs. It may be argued that bringing back the COS in a consultation round among the original online panel of participants after the consensus meeting therefore is not necessary. However, the evidence regarding this issue is still limited to the findings in two procedures.

Bringing back the COS for confirmation after the consensus meeting is an easy step and can potentially add to the face validity and uptake of the final COS, because the knowledge and expertise of the larger online participant group is used and bias is avoided. The evidence base for the value of a validation round needs to be enlarged and guidelines should be made on how to perform the consultation round. We propose doing this in further consensus procedures.

Data availability statement

Data sharing not applicable as no datasets generated and/or analysed for this study. The datasets used and/or analysed during the COSGROVE Study will be held by the COSGROVE team at patricia.healy@nuigalway.ie. All data relevant to the DCOHG Study are included in the original article or uploaded as supplemental information in the original article.

Ethics statementsPatient consent for publication

Not required.

Ethics approval

This study involves human participants. COSGROVE: ethical approval for the study was obtained from the Medical Ethics Review Committee of the University of Groningen (reference number: METc 2016.660). DCOHG: a waiver for ethical approval was obtained from the institutional review board of the Academic Medical Center Amsterdam (reference number E2-172, 11-05-2016). Participants gave informed consent to participate in the study before taking part.

Acknowledgments

P Williamson, Liverpool Clinical Trials Centre, University of Liverpool, a member of the Liverpool Health Partners, Liverpool, UK, and all authors in the COSGROVE Study group and the DCOHG Study group: A Baschat, Johns Hopkins Center for Fetal Therapy, Baltimore, USA; F Bloomfield, Liggins Institute, University of Auckland, Auckland, New Zealand; M Daly, Advocacy and Policymaking, Irish Neonatal Health Alliance, Wicklow, Ireland; D Devane, Health Research Board-Trials Methodology Research Network, Galway, Ireland; L Kenny, The Irish Centre for Fetal and Neonatal Translational Research (INFANT), University College Cork, Cork, Ireland; A Khalil, Fetal Medicine Unit, St George’s University and St George’s University Hospitals NHS Foundation Trust, London, UK; A Papageorghiou, Fetal Medicine Unit, St George’s University and St George’s University Hospitals NHS Foundation Trust, London, UK; C Dean, Pregnancy Sickness Support, UK; J Duffy, Institute for Women’s Health, University College London, London, UK; N Gauw, Dutch hyperemesis gravidarum patient foundation (ZEHG), The Netherlands; B Goes, Leiden University Medical Center, Leiden, The Netherlands; J van 't Hooft, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, The Netherlands; M Koot, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, The Netherlands; J Rodenburg, General Practice Czaar Peter Medical Center, Amsterdam, The Netherlands; T Roseboom, Amsterdam University Medical Centers, University of Amsterdam, Amsterdam, The Netherlands.