The aim of our Delphi study was to construct and validate a performance measurement framework specifically designed for telephone triage services. We employed the Delphi method to identify and filter the most crucial PIs for our measurement framework. Concurrently, we used the three-round NGT to establish consensus among experts.

The Delphi method facilitates structured expert opinion integration, capitalising on its strengths in eliciting informed inputs and fostering consensus-building. Chosen for its robust consensus-building capability, the NGT method ensures balanced participation from all group members, making it particularly effective in scenarios necessitating decision-making, complex problem-solving, prioritisation, and consensus achievement. [11, 12]. The NGT method has found applications in research aimed at generating ideas, solving problems, and establishing priorities [13, 14]. Additionally, the technique is valued for its rapid implementation, transparent process, inclusivity, and ease of replication [14,15,16].

We combined NGT-led group discussions with a digital survey. The digital survey was particularly useful given the geographical dispersion of NGT group participants, facilitating a time-efficient and inductive exploration of the subject, free from external bias [13, 14]. The NGT process unfolds in two main phases: an initial ranking, where participants evaluate the PIs using a Likert scale, and a subsequent re-ranking, allowing participants to revise their initial evaluations based on insights from a secondary survey [12].

When illness symptoms appear, patients can reach out to outpatient healthcare professionals at their local health centre during weekdays for non-urgent conditions. Since 2018, a 24/7 Medical helpline 116,117 has been available in Finland. The MH is accessible throughout Finland, with the exception of Lapland and the Åland Islands. Wellbeing services counties are responsible for organising the service in their area.

After hours, on weekends, and on public holidays, patients are advised to call MH before going to the ED. In case of an emergency, critical illness, or injury, patients need to call the emergency number 112. At the MH, nurses conduct telephone triage using a protocol that follows the national principles of urgent care coordinated by the Ministry of Social Affairs and Health. Ilkka [17] This protocol includes a six-tier urgency classification system. The nurses guide patients to the most appropriate care based on their needs or, in an emergency, forward the call to 112. Calls are documented in the electronic health record (EHR) using the ISBAR protocol. The acronym ISBAR stands for identify, situation, background, assessment, and recommendation [18]. Reasons for encounters are documented using the ICPC-2 classification [19]. Nationally, it is required to document at least the patient's need for care, medical history, status, and plan, i.e., the outcome of the triage. Despite recommendations for call documentation, there is significant variation in the quality and comprehensiveness of documentation among professionals [20, 21].

The participant experts in our study were chosen through purposeful sampling. This method was strategically employed to engage experts with in-depth knowledge and vast experience in telephone triage, thereby ensuring the inclusion of information-rich cases. The primary aim of this sampling technique was to facilitate a comparative analysis of expert opinions, allowing us to discern both similarities and differences in their perspectives and insights [22, 23]. The NGT group consisted of chief physicians from all 20 central hospitals in Finland, representing the full geographical spread of emergency services across the country. Each participant specialised in Emergency medicine and/or Anesthesiology and intensive care medicine. Additionally, they were responsible for the operations of the MH, as well as both primary and specialised care ED services within their respective regions. Each respondent also held a leadership position, with oversight and reporting responsibilities for the entirety of urgent care services. This deliberate selection of experts was essential to identify potential regional variations and achieve consensus.

Ten experts participated in the NGT group activities. After agreeing to participate, experts received an email link to access the survey, accompanied by three reminder emails for each round. To accommodate participants' schedules, we extended response deadlines by 11 days for the second round and by one month for the third. Questionnaires were distributed via the Webropol© platform, with the introductory page of each round providing detailed instructions and guidance on completing the questionnaire. Following each round, experts received a summary of the results from the preceding round. This iterative process aimed to achieve the highest possible level of consensus and to support participants' thinking towards the next round.

While the first round of a classical Delphi process is typically unstructured, we chose to structure this phase based on our understanding of essential indicators for the framework, thereby optimising professional time, as suggested by Rowe et al. (1991). In this round, our expert panel evaluated 28 PIs organised into five distinct performance dimensions. Each PI was rated on a 10-point Likert scale to assess its importance and relevance. Our Delphi process focused on validating these PIs while ensuring their balanced distribution across the framework. To maintain clarity and simplicity, we restricted the number of PIs to two per performance dimension. The decision was made to build a balanced measurement framework, which is common in measurement frameworks for management (e.g. Balanced Scorecard, Quadruple Aim). We also aimed to enhance its practical application in management by choosing a limited number of measures and thus reducing information overload [24]. At the end of the first round, we selected two PIs from each dimension that received the highest scores to proceed to the next round. In cases where two PIs received identical scores, we chose the indicator that had a higher score in terms of relevance for inclusion.

The duration for submitting responses in the second round was extended from December 29th, 2021, to January 12th, 2022. Consequently, due to the low rate of response, the deadline was further extended to January 23rd, 2022. Additionally, three reminder emails were dispatched. In this round, respondents appraised the accuracy and sensitivity of the PIs using the same 10-point scale. They assessed how well each indicator represented its assigned performance dimension, with scores below seven prompting suggestions for more accurate alternatives.

Following the second round, we conducted a workshop with the authors of the study to analyse the responses and refine the framework for the final round. In particular, we scrutinised PIs that received a rating of less than 7 from any evaluation perspective, seeking alternative indicators based on open-ended responses. The primary objective was to enhance the framework's effectiveness in advancing the development of telephone triage services and to maximise its strategic value in both short-term and long-term scenarios.

The response period for the third round was set from February 13th, 2022, to March 3rd, 2022. In response to the low response rates observed during the first three rounds and to further confirm the utility of our framework, a final performance framework validation round was conducted from October 29th, 2023, to November 7th, 2023. The final evaluation in this round included assessing the framework's contribution to the development of telephone triage services, its utility in aiding both short-term and long-term strategic development, and its overall relevance. Crucially, participants also evaluated the balance of the framework, providing insights into its strengths and limitations and offering open-ended feedback. Upon completion of each round, we entered the data into MS Excel© for analysis. We conducted a descriptive analysis, which included calculating frequency, percentage agreement, mean (indicator scores), and median. A flowchart (Fig. 1) provides an overview of the research process.

A Flowchart of the research process

Members of the Delphi panel were informed in their invitation email and related materials that they were free to withdraw from the study at any point. Their agreement to participate was confirmed via email, where we emphasised the entirely voluntary nature of their participation. To ensure confidentiality, the names and contact details of the participating professionals were securely stored in a restricted-access location, separate from the main data repository. Due to the study's nature, obtaining institutional review board approval was not required. The University of Helsinki, Finland, coordinates the study. The study adheres to the ethical principles outlined in the Helsinki Declaration.