External validation of the Manchester Acute Coronary Syndromes ECG risk model within a pre-hospital setting

WHAT IS ALREADY KNOWN ON THIS TOPIC

The Manchester Acute Coronary Syndromes ECG (MACS-ECG) prediction model has been derived and validated to identify non-ST elevation myocardial infarction (NSTEMI) in patients with acute chest pain in the emergency department.

MACS-ECG has similar diagnostic accuracy to an emergency physician.

MACS-ECG uses objective parameters without requiring subjective ECG interpretation, which is an advantage for use in the prehospital environment.

WHAT THIS STUDY ADDS

In this secondary analysis of the PRESTO data, MACS-ECG had a relatively low c-index and showed very low sensitivity at both the pre-determined and optimum cut-off in the prehospital environment.

Paramedic ECG interpretation had superior sensitivity but lower specificity.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

Future research should focus on using different methods to extract digital ECG data and derive prediction models that can detect NSTEMI, for example, by using deep learning with a large dataset. Focusing on paramedic training and other diagnostic information, such as point of care troponin testing, may have a greater yield.

Background

Chest pain is one of the most common reasons for an emergency ambulance to be requested.1 2 A diagnosis of acute myocardial infarction (AMI) is often suspected from the clinical history. In such cases, the ECG is the first-line investigation and should be recorded in the pre-hospital environment.3 This will identify an ST elevation myocardial infarction (STEMI), which is indicative of acute coronary occlusion, in some patients. Patients with STEMI require immediate revascularisation and should be transported to a heart attack centre.

All other patients with suspected AMI (non-ST elevation AMI (NSTEMI)) will require transport to hospital for further diagnostic tests. However, only a minority of these patients have NSTEMI.4 Most patients do not require inpatient treatment. There is, therefore, great potential to reduce unnecessary transport to hospital by improving pre-hospital diagnostics for NSTEMI.

In hospital emergency departments (EDs), decision aids such as the HEART (History, ECG, Age, Risk factors, Troponin) score or Troponin-only Manchester Acute Coronary Syndromes (T-MACS) are widely used to rapidly identify patients in whom the diagnosis of NSTEMI can be safely ‘ruled out’.5 6 There has been growing interest in deploying such decision aids in the pre-hospital environment. One barrier to implementation is the requirement for paramedics to interpret ECGs for signs of acute ischaemia (other than STEMI). This is not currently a routine element of clinical practice for paramedics.

The Manchester Acute Coronary Syndromes ECG (MACS-ECG) prediction model was recently derived and externally validated within an in-hospital setting (table 1).7 MACS-ECG uses objective ECG measurements to calculate a score, which can be used to determine the probability of NSTEMI. At a cut-off of 27.4% calculated probability, MACS-ECG had a specificity of 95.2% and a sensitivity of 23.5% for NSTEMI when externally validated.7 This is similar to the accuracy obtained with subjective ECG interpretation by emergency physicians. For example, in a validation study of the HEART score, we calculated that the sensitivity of significant ST depression (as interpreted by the treating emergency physician) was 20.0% for acute coronary syndrome (ACS), whereas the specificity was 97.2%.8 However, the predictive accuracy of this model within a pre-hospital setting is unknown.

Table 1

Variables included in the MACS-ECG model

MACS-ECG may have particular advantages in the pre-hospital setting. Paramedics are generally trained to recognise STEMI but are not routinely trained to detect ECG signs of NSTEMI. Avoiding the need for subjective interpretation could therefore facilitate pre-hospital diagnostics, for example, alongside point of care troponin testing or by identifying high-risk patients who may benefit from immediate transport to a heart attack centre, bypassing local hospitals. This would reduce the need for secondary ambulance transfer for patients requiring coronary intervention, an approach that has been shown to reduce time to angiography.9 10 At present, calculating the output of the MACS-ECG model requires manual measurements (of T wave height and ST depression). However, if shown to be sufficiently accurate for clinical use, it could be incorporated in ECG software applications to produce automated interpretation for possible NSTEMI.

Therefore, we aimed to evaluate the diagnostic accuracy of the MACS-ECG prediction model in the pre-hospital environment. Next, we aimed to compare the accuracy of the MACS-ECG prediction model to that of paramedic ECG interpretation.

MethodsStudy design and setting

This is a secondary analysis of the Pre-hospital Evaluation of Sensitive Troponin (PRESTO) study data.11 The PRESTO study was a multi-centre prospective diagnostic test accuracy study taking place at 12 hospitals and 4 NHS ambulance trusts in the UK (listed in the online supplemental appendix 1) and funded by the National Institute for Health and Care Research for Patient Benefit scheme. Its primary objective was to evaluate the diagnostic accuracy of decision aids (T-MACS and the HEART score) used with point of care cardiac troponin assays for identifying AMI in the pre-hospital setting. Data were collected between 26 February 2019 and 23 March 2020. The study received ethical approval from the East of Scotland Research Ethics Service (reference 18/ES/0101) and was prospectively registered at clinicaltrials.gov (reference NCT03561051).

Participants

We included adult patients aged >18 years who received an emergency ambulance response for a primary complaint of chest pain in whom participating paramedics suspected the diagnosis of an ACS. We excluded participants who had STEMI on the pre-hospital ECG, those who had no pre-hospital ECG available for analysis and participants who had an uninterpretable ECG. All participants provided initial verbal consent in the pre-hospital environment. Full written informed consent was then sought in hospital or once the patient had been discharged home. We excluded participants who declined to provide written informed consent and those who could not be contacted to obtain written consent.

Paramedics who participated in screening and data collection during the PRESTO study were provided with bespoke training. This included training in the fundamentals of Good Clinical Practice, study protocol training and training in the interpretation of ECGs for signs of acute ischaemia. Training was made available in both face-to-face and online formats.

Data collection

Paramedics recorded ECGs as part of the routine clinical care they provided for patients with suspected ACS, prior to transporting patients to hospital. The ECGs were then uploaded to the electronic case report form environment (Castor EDC). Paramedics interpreted the ECGs during the pre-hospital phase of patient care. They were asked to interpret each ECG as ‘normal’ or ‘abnormal’ and to specifically note the presence or absence of left bundle branch block (LBBB), abnormal T wave inversion and ST depression. At the time of interpretation, paramedics were blinded to all biomarker results (all troponin testing, including point of care troponin testing, was undertaken in the hospital) and, because of the timing of interpretation, to patient outcome.

ECG data for calculation of the output of the MACS-ECG model were extracted using a bespoke digital calliper.12 EP Callipers is a computer programme designed to measure electronic ECG for heart rate, voltage and ECG changes in millimetres.12 The software is not automated. Researchers must select two points on the ECG (a reference point and the point of interest) and the software will measure the distance between those. First, measurements were calibrated against the original ECG calibration waveform. Second, measurements were taken to the nearest 0.5 mm using digital callipers. The primary researcher extracted ECG data. A sample of approximately 15% of all ECGs, including all ECGs with imperfect resolution, were also checked by a second investigator. Any discrepancies were resolved by discussion. All participants with ECGs that had insufficient quality to allow clear measurement of the required parameters were excluded from this analysis.

Researchers measured all ECG parameters required for calculation of the MACS-ECG prediction model output including the height of the T wave in V1, the presence or absence of Wellen’s syndrome type A, the mean ST depression in leads II and III, the mean ST depression in leads V2 and V3 and the mean ST depression in leads V5 and V6. In accordance with the original study, ST depression was measured 80 ms after the J point.

Using a bespoke case report form, participating paramedics also collected data on patient demographics, the time and date of the ambulance response, past medical history and physical observations.

Outcomes

The primary outcome (or target condition) was a diagnosis of type 1 AMI. The diagnosis of AMI was adjudicated by two investigators acting independently (EC and JC). AMI was adjudicated in accordance with the fourth universal definition of myocardial infarction, which requires a rise and/or fall of cardiac troponin with at least one concentration above the 99th percentile upper reference limit, combined with at least one of the following: symptoms of myocardial ischaemia, new ischaemic echocardiogram changes, pathological Q wave or ECG changes.13 All patients were transported to hospital and underwent laboratory troponin testing in accordance with contemporary national and/or international guidance, which formed the reference standard investigations for AMI.

Secondary outcomes included major adverse cardiac events (MACEs). MACE was defined as the occurrence of death (all cause), coronary revascularisation or incident AMI within 30 days.

Sample size

The sample size for the PRESTO study was calculated assuming that the prevalence of the primary outcome would be approximately 10%, assuming that the index test evaluated had a specificity of approximately 45% and assuming that the index test would achieve 100% sensitivity. We also accounted for 5%–10% of patients having missing data. This would require a total sample size of 700 participants. This sample size calculation applied to the primary analyses for the PRESTO study. As this is a secondary analysis of data from PRESTO, no formal sample size calculation was performed for this work.

Statistical methods

Continuous data were summarised using mean and SD, while categorical variables were summarised using frequencies and percentages. We summarised the data both as a whole cohort and across subgroups of those who did and did not have the primary outcome.

Predictive performance of the model was quantified using calibration (agreement between the observed and expected event proportions) and discrimination (ability of the model to differentiate those who had the event from those who did not). For calibration, we produced calibration plots. For discrimination, we constructed a receiver operating characteristic (ROC) curve, with this summarised using the C-index was calculated with 95% CIs (where a C-index of 0.5 indicates discrimination no better than change, and values closer to 1 being better discrimination). Additionally, we calculated test characteristics (sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV)) with respective 95% CIs using the cut-off defined in the original derivation study for MACS-ECG (27.44% probability). In an exploratory analysis, we also proceeded to calculate test characteristics at the cut-off with maximum sensitivity for AMI. Finally, we calculated the sensitivity, specificity, PPV and NPV of paramedic ECG interpretation (normal vs abnormal ECG) for type 1 AMI.

Statistical analyses were undertaken using SPSS, V.27.0 (SPSS) except for calculation of sensitivity, and specificity, for which we used MedCalc software online.14

Results

A total of 817 patients were included in the original PRESTO study, of which 691 were eligible for inclusion in the primary analysis of validating MACS-ECG for type 1 AMI prediction and 687 were eligible in the analysis of paramedic’s ECG analysis accuracy for type 1 AMI prediction (figure 1). Baseline characteristics of the included participants are reported in table 2. Among the 691 participants, 87 (12.6%) patients had an adjudicated diagnosis of type 1 AMI.

Figure 1Figure 1Figure 1

Flow diagram of patient’s inclusion. ACS, acute coronary syndrome; AMI, acute myocardial infarction.

Table 2

Summary of baseline characteristics

Using the originally derived cut-off (27.4% calculated probability of NSTEMI), MACS-ECG had a C-index of 0.68 (95% CI: 0.61 to 0.75). The diagnostic performance for prevalent AMI is summarised in tables 3 and 4. The model had low sensitivity of 2.3% (95% CI: 0.3% to 8.1%) and high specificity 99.5% (95% CI: 98.6% to 99.9%). In an exploratory analysis, we identified that the ROC-optimised cut-off to maximise the sensitivity of MACS-ECG was at a probability of 9.56% for NSTSEMI. The test characteristics at that optimised cut-off are shown in table 4. The optimised model showed a sensitivity of 50.6% (95% CI: 39.6% to 61.5%) and specificity of 83.1% (95% CI: 79.9% to 86.0%).

Table 3

Proportion of patients with AMI in validation of the MACS-ECG

Table 4

Diagnostic performance of MACS-ECG

By categorising pre-hospital ECGs as ‘normal’ or ‘abnormal’, paramedic ECG interpretation had a sensitivity of 71.3% (95% CI: 60.7% to 80.5%) and specificity of 53.8% (95% CI: 53.8% to 61.8%). The diagnostic performance of paramedic ECG analysis is summarised in tables 5 and 6.

Table 5

2×2 table for the diagnostic accuracy of paramedic ECG interpretation (‘normal’ vs ‘abnormal’) for NSTEMI

Table 6

Test characteristics showing the diagnostic accuracy of paramedic ECG interpretation (‘normal’ vs ‘abnormal’) for NSTEMI

We further evaluated the accuracy of paramedic interpretation of specific ECG parameters (including ST elevation, ST depression, T wave inversion and LBBB). Those findings are reported in online supplemental appendix 2. In general, paramedics achieved high specificity when interpreting these parameters, with sensitivity varying between 10.3% (for ST elevation), 5.7% (for LBBB), 33.3% (for ST depression) and 40% (for T wave inversion).

A calibration plot for the MACS-ECG model is shown in online supplemental appendix 3. This demonstrates systematic underprediction of risk with a gradient of 0.0628 (whereas a perfectly calibrated model will have a gradient of 1) with an intercept of 0.0852. Only six probability deciles could be created for the calibration plot because over 80% of participants had no abnormalities on ECG and thus had the same expected probability of AMI.

Discussion

In this paper, we have evaluated the validity of the MACS-ECG model for pre-hospital detection of NSTEMI in chest pain patients with suspected ACS. The MACS-ECG model showed very low sensitivity and had a relatively low C-index. Even identifying the cut-off for predicted probability (calculated by MACS-ECG) that had maximum sensitivity would not yield a satisfactory balance between sensitivity and specificity to inform clinical decision making. However, paramedic ECG interpretation had superior sensitivity in this study. Specificity was lower, though this may reflect the nature of the question (‘normal’ vs ‘abnormal’ ECG). Of the two approaches, it appears that paramedic interpretation of an ECG as ‘abnormal’ is more likely to add clinical value given the extremely low sensitivity of the MACS-ECG model.

The MACS-ECG model showed higher sensitivity during the initial derivation study than in this external validation study, reflecting that this study focused on the performance within a pre-hospital setting. In the derivation, the model had a sensitivity of 25.6% and specificity of 96.3%.7 It is unclear why there was such a discrepancy in sensitivity between studies, though there are many differences between hospital and pre-hospital environments. Pre-hospital ECGs may, for example, have been more subject to movement artefact or the precision of the ECG machines could be different.

The optimal sensitivity, specificity, PPV and NPV for an ECG prediction model are hard to specify and will depend on how the ECG interpretation is to be used. It is unlikely that the ECG alone could ever be used to rule out AMI and accounting for other information, such as cardiac troponin concentrations, is likely to be important. In this study, we hoped to show that MACS-ECG would be at least no worse than paramedic ECG interpretation. Had that been proven, it may have been possible to replace the need for subjective interpretation (and with it the ongoing training requirements) with an automated calculation. However, our findings clearly suggest that paramedic ECG interpretation is superior to use of the MACS-ECG model in the pre-hospital environment.

Focusing future efforts on enhancing paramedic training may add greater value than continued attempts to implement automated systems for interpretation. A Canadian study investigated the sensitivity of paramedics detecting STEMI in pre-hospital environment before and after taking an ECG interpretation course.15 Prior to the additional training, paramedics had a sensitivity of 78%. This increased to 99% with a specificity of 68% after 21 hours of training.15 Another study investigated the feasibility of improving NSTEMI identification in the pre-hospital environment by sending electronic copies of ECGs from paramedics to cardiologists; this demonstrated a 32% (95% CI: 14% to 55%) sensitivity for the NSTEMI cases that were identified by the cardiologist and patients were transferred directly to PCI.16 Taken in conjunction with our findings, these data suggest that pre-hospital ECG diagnosis of NSTEMI is likely to be challenged by low sensitivity, regardless of how the ECG is interpreted.

Strengths and limitations

We collected data prospectively and included a comparison with paramedic interpretation. In this study, patients were subjected to robust reference standard testing for AMI in hospital. Our study does have some limitations; for example, this is a substudy nested within PRESTO, meaning that our sample size was calculated for the primary objective of the main study and not specifically for this analysis. It is also possible that paramedics who recruited to the study were more confident in ECG interpretation than paramedics who did not enrol patients. Finally, because of the challenges of working in the pre-hospital environment with unstable patients, clinical urgency and variable environmental conditions, 49 ECGs were excluded from our analysis due to poor recording quality or unsatisfactory resolution. This may reflect differences in the hardware available for recording ECGs, movement artefact or artefact introduced when uploading ECGs to the electronic case report form. Unfortunately, it is an unavoidable limitation of research in this environment and therefore our study likely represents the best possible evaluation of the MACS-ECG prediction model in real-world pre-hospital practice.

Future research

If we had demonstrated that MACS-ECG had similar or better diagnostic accuracy to subjective interpretation by paramedics, then we could have incorporated it within decision aids (eg, T-MACS or the HEART score) to provide an objective method to extract diagnostic information from the ECG. Evaluating the accuracy of T-MACS and the HEART score incorporating the output of MACS-ECG in the place of subjective ECG interpretation may be a valuable goal for future work. However, with such low sensitivity it seems unlikely that MACS-ECG could be used in that manner.

It may therefore be wise for future research to focus on refining the MACS-ECG model or using different methods to derive prediction models that can detect NSTEMI, for example, by using more granular digital data extracted from ECG images and applying techniques such as deep learning with large datasets. Also, previous research has shown that paramedics are able to improve their identification of STEMI after several hours of additional training15; therefore, future work should evaluate if the same is true for identification of NSTEMI.

Conclusion

We found that the MACS-ECG prediction model has very low sensitivity and high specificity for identifying NSTEMI in the pre-hospital environment. Subjective ECG interpretation by paramedics had higher sensitivity but lower specificity. Neither approach had a sufficiently high PPV or NPV to ‘rule in’ or ‘rule out’ NSTEMI alone. Future work should evaluate their value alongside other information, for example, as part of a validated decision aid.

Data availability statement

The data that support the findings of this study are available from the corresponding author, AA, upon reasonable request.

Ethics statementsPatient consent for publicationEthics approval

The study received ethical approval from the East of Scotland Research Ethics Service (reference 18/ES/0101).

留言 (0)

沒有登入
gif