A modified Delphi approach [10] was used. The Delphi method is an iterative process of repeated questionnaires that are bundled with the results of the previous rounds to a panel of experts with the goal of finding consensus. The items to be assessed in the Delphi rounds were gathered in initial interviews with the experts in round one, and from the published literature. The Delphi approach has successfully been used when seeking consensus in emergency care [11], and when assessing questions related to triage [12]

To create the expert panel, three important groups were identified: Clinicians working in emergency care, researchers publishing studies related to triage, and designers of ED priority triage systems. To the clinical part of the expert group, we recruited Swedish physicians and registered nurses with a specific interest and knowledge of triage. To the research group, we recruited individuals who were listed as either the first, second or last author of a published study during the last five years related to emergency priority triage and outcomes for adult patients. Lastly, in the designer group we recruited designers or editors of emergency care priority triage systems and persons medically responsible or integral in the implementation of such systems. Representatives from all major triage systems in use in Sweden (RETTS, SATS, WEST) were recruited. The experts in all three groups were recruited from geographically diverse locations in Sweden. If an expert missed any of the questionnaire rounds, they were not excluded from joining the following rounds (Table 1).

The data collection was split into collecting suggested outcomes (round one) during three weeks in the spring of 2021, and testing of these outcomes by a Delphi approach (round two to four) during the last months of 2021. The experts were informed that the term outcome could include any measurable outcome that they felt could be relevant to evaluate triage, including e.g. admission to in-hospital care, diagnosis, interventions or laboratory testing.

The outcomes to be assessed were gathered through qualitative semi-structured interviews using written notes shared with the expert. During the interview, the experts suggested outcomes and assigned them to either of two groups: Outcomes that would motivate red priority at the ED, i.e. with resuscitation team activation, and outcomes with no need for resuscitation team activation but that are still considered time-critical; i.e. that would motivate red or orange priority. After round one the experts could not suggest new outcomes.

For every proposed outcome the experts were asked to also suggest a time-frame within which the outcome should be evaluated. When the suggested time-frames for the outcome varied among the experts, a question of the time-frame was included separately in the Delphi questionnaire, see Fig. 1. If only similar time-frames were suggested, the time-frame was embedded in the outcome description, exemplified in Fig. 2. Two groups of time-frames emerged from the interviews; short-term, often described as”in the ED'' or up to a day from leaving the ED, and long-term, described as a couple of days up to 30 days. Since no clear cutoffs were presented in the interviews, we included this as a question at the beginning of the questionnaire, intentionally leaving the cut offs as overlapping, see Table 2.

Outcome question without a specified time-frame, followed by a question regarding time-frame

Outcome question with the time-frame included in the definition of the outcome

At the end of each interview the suggested outcomes were repeated back to the experts to confirm that they were understood correctly. The interviews were conducted during the spring of 2021 and held through video calls due to the COVID-19 pandemic. Two pilot interviews were conducted with emergency care specialist nurses before the first interview, which led to changes in how the questions were presented, to avoid misunderstandings.

The Delphi questionnaire was constructed and answered through REDCap which is a web-based platform that provides secure, web-based access to research data and tools to gather it through questionnaires [13, 14]. Our questionnaire was written in Swedish and piloted on the above mentioned specialist nurses before being sent to the experts.

Subsequent rounds followed a modified Delphi process as described by Clayton [10]. Experts were invited to the Delphi questionnaire via individual emails sent through REDCap. The outcomes gathered in round one were presented to the experts together with the experts' arguments for or against the outcomes. Potential for conflict between different outcomes were described next to the affected outcome, such as that positive consensus of one outcome could make another redundant. Admission to ward was added as an outcome in round two based on previous research; the inclusion of this outcome was planned beforehand. The experts were informed that outcomes could be added by the researchers, but not of which specific outcomes.

All outcomes were presented as statements with a similar structure including both outcome and priority level as seen in Figs. 1 and 2. The outcomes were assessed via a five-stepped Likert scale from “Strongly agree” to “Strongly disagree”. The experts also had the possibility, through a free text input, to supplement their opinion with new arguments that they believed were missing. These arguments were analyzed with manifest qualitative content analysis [15] and arguments for or against outcomes were presented verbatim alongside the outcomes in the following rounds. From round three, the aggregated expert opinions of the previous rounds were displayed alongside the outcome, and from round four this included stability. All questionnaires were sent out with three reminders and the experts had one month to answer each questionnaire.

The data from the Delphi questionnaires were extracted from REDCap and analyzed in Microsoft Excel 2013 for Windows. The predetermined cutoff for consensus was that 70% of the responses fell in either of the upper two alternatives of the five-step Likert scale [16], i.e. positive consensus, or in the lower two alternatives, i.e. negative consensus. Stability was similarly deemed reached if 70% of the answers did not change from its dichotomized group in the previous round, i.e. positive or negative. If an outcome reached the predetermined cutoffs for consensus and stability, it was excluded from further rounds. This could happen at earliest after round three since stability required two rounds of questionnaires to be calculated. For the time-frames, consensus/stability was calculated per suggested cutoff in each group (long term/short term).

Since all outcomes that gained consensus also reached stability it was decided to present median and interquartile range [IQR] for all outcomes based on measures of spread [17]. The round where consensus was reached was recorded together with the median and IQR from the round when both consensus and stability were reached.