The ATT was developed by a stepwise approach including the [1] review of existing literature regarding patient safety and areas of risk for AEs in the prehospital field [2] expert panel discussions with adaptation of the ATT through a video link, and [3] clinical evaluation of the ATT through RRR (Fig. 1).

Flow chart of the development process of the EMS trigger tool. EMS, emergency medical services. RRR, retrospective record review

The EMS system in Sweden is funded by taxes across regions, resulting in local variations in guidelines and documentation systems. At a national level, it was specified that in each ambulance, one of the two EMS clinicians must be a registered nurse (RN) often with a degree of master in ambulance/intensive/anaesthesiologic care.

We invited the majority of EMS organizations and three services were recruited across Sweden: an urban service with 85,000 EMS missions each year and a median mission time of 70 min and two urban–rural mixed services with 53,000 and 41,000 EMS missions each year and median mission times of 78 and 66 min, respectively. Two of these services participated in the first two sessions of structured discussions and the other one participated in the third.

We used the World Health Organization (WHO) terminology for incidents including NM, NHI, and HI [15, 16] and categorized the incidents according to the National Coordination Council for Medication Error Reporting and Prevention index (NCC MERP) [17].

An NM incident (e.g. lack of documentation) neither affects nor harms the patient but poses the risk of an error and is categorized as (A) or (B) according to the NCC MERP.

An NHI (e.g. omission of electrocardiogram (ECG) in patient with chest discomfort not diagnosed with acute coronary syndrome) affects the patient but does not cause harm. It is categorized as (C) ‘An incident that affected the patient but did not cause any harm’ or (D) ‘An incident that affected the patient and demanded observation or treatment to assure that no harm occurred’.

NMs and NHIs are traditionally not used when using TTs but could be valuable in terms of evaluating the clinical setting by receiving information about commonly occurring NMs and NHIs [18].

An HI (adverse event; e.g. omission of ECG in patient with chest discomfort and later diagnosed with ST-elevated myocardial infarction, thus delaying time to causal treatment) harms the patient. It is categorized as (E) ‘Contributed to or resulted in temporary harm and required intervention’; (F) ‘Contributed to or resulted in temporary harm requiring outpatient care, readmission, or prolonged hospital care’; (G) ‘Contributed to or caused permanent patient harm’; (H) ‘An event that required lifesaving intervention within 60 min’; or (I) ‘Contributed to the patient’s death’.

Existing TTs for inpatients, homecare, paediatric care, and psychiatric care presented by the Swedish Association of Local Authorities and Regions (SKR) were reviewed for developing the new ATT [19]. To cover a prehospital context and to identify risk areas for patient safety, PubMed, Cinahl and Medline was searched with the following keywords: patient safety, prehospital, ambulance, trigger tool, and adverse event. Headlines and abstracts were read and included with the following criteria (1) a description of incidents regarding patient safety in EMS measured with various methods, (2) ≥ 18 years of age, (3) not published before 1980.

Five experts, including RNs and medical doctors (MDs), were invited via email to contribute during the discussions. The RNs were eligible to participate in the development of the ATT if they had 10 years’ EMS clinical experience, previous experience in patient safety in an EMS organization, and experience regarding the usage of a TT. The MDs were recruited because of their knowledge and experience in the methodology of developing TTs. They were also responsible for patient safety in the healthcare system at a regional or national level within their organization. All members of the panel where confident in other methods used in the patient safety context such as incident reporting or root cause analysis. The experts were recruited on the authors opinions who was suitable but also depending on personal interest from the experts.

Three sessions of structured discussions was employed to reach a consensus among a panel of experts regarding the ATT, where the collective opinion of the group is deemed stronger than that of each individual [20]. An expert is known to be a specialist in their field, an individual who has knowledge on a specific matter [21, 22]. During the three sessions, the discussions was structured as follows: each definition and trigger were analysed using a shared screen view for language, clinical relevance, and user-friendliness. The document was seen by all experts, and the removal, adaptation, or combination of triggers was done instantly. Each correction was approved by the experts before moving on to the next trigger. The session was video recorded so that the session would be viewable again and the corrections were emailed to the experts with the ability to give written feedback. Consensus was considered achieved when all experts had approved the ATT via email, and no further feedback was provided by them in writing.

The measurement of patient safety with TTs requires access to patient records. In this light, review teams consisting of an RN and an MD were formed, where the RN starts with a primary review of the patient record with guidance of a list of triggers. A trigger can be a clue that an incident has occurred. Each trigger comes with a definition containing criteria that need to be fulfilled for the trigger to be considered positive.

The RN searches the records for positive triggers, reviews them for a potential incident, and classifies the incident according to the three first steps (AB, C, D) of the NCC MERP [17]. Records with an incident graded C and D undergo a secondary review by the MD, who assesses them for an HI. If an HI is found, the MD classifies it according to the last five steps (E, F, G, H, I) of the NCC MERP. The MD also classifies the type of HI and whether it is preventable by using the following scale: (1) ‘not preventable’; (2) ‘probably not preventable’; (3) ‘probably preventable’, and (4) ‘certainly preventable’.

The three EMS organizations each formed a review team consisting of a MD and a RN. The MD was in medical charge of their organization, and the RN had extensive experience of working in the EMS within this organization. One organization used two RNs having comparable experience in working in the EMS in the primary review.

In a pilot study before session two of the structured discussions, two review teams from two different organizations randomized 150 records and performed an RRR to gain experience in using the ATT.

Before session three of the structured discussions, three review teams from different organizations randomized 900 records and performed an RRR to further evaluate the ATT and to receive frequencies of positive triggers and incidents from their organizations. Before the RRR the review teams received training in how to extract and randomize data, how to use the ATT within Microsoft Access®, definitions of common terminology regarding a TT (e.g. positive trigger, NM, NHI, and HI), and examples with fictitious records. The reviewers had no previous experience in using a TT.

The criteria for inclusion in both RRR was age ≥ 18 years and a primary mission where a patient assessment takes place. Children were excluded in this study because several studies have shown that different triggers are required to study the paediatric population in EMS [23, 24]. There are plans to develop a set of triggers adapted for children.

The positive triggers, NMs, NHIs, and HIs were presented in a frequency-based manner according to both the WHO and the NCC MERP. Each member of the review teams graded the triggers according to clinical relevance, comprehensibility, and utility with a 4-point Likert scale, where 1 = not relevant, 2 = somewhat relevant, 3 = quite relevant, and 4 = highly relevant after the RRR. The item-level content validity index (I-CVI) was calculated for each trigger by summing the number of reviewers grading trigger 3 or 4 and divided by the total number of reviewers. I-CVI of 0.80 or higher was considered highly relevant [25]. The positive predictive value (PPV) was calculated for each trigger by how many times the trigger resulted in a near miss, no harmful or harmful incident divided by the total times the trigger was found multiplied by 100 [26]. The Mersenne Twister algorithm was used for randomizing the records [27].

Two sessions of independent inter-rater reliability (IRR) tests between two primary reviewers were conducted, with the triggers serving as the variable for testing. The total outcome of positive triggers was summed up and analysed in a confusion matrix with Cohen’s kappa (Table 3) [28]. Kappa values of 0.21–0.40, 0.41–0.60, 0.61–0.80, and 0.81–1.00 were respectively considered fair, moderate, substantial, and almost perfect [29]. The ATT produces an overrepresentation of negative triggers when no incidents have occurred and creates a prevalence problem which causes Cohen’s kappa to be low [28, 30]. Therefore, Cohen’s kappa was complemented with a prevalence-adjusted and bias-adjusted kappa (PABAK) [30]. The interpretation of PABAK and Cohen’s kappa is the same [31]. All analyses were performed with R studio version 2023.03.0 + 386.