Introduction

Mechanical thrombectomy has become the standard of care for patients with acute ischaemic stroke due to a proximal anterior circulation large vessel occlusion.1 2 A major determinant of the clinical outcome following mechanical thrombectomy is the time to recanalisation.3 4 Analysis of imaging data by artificial intelligence (AI)-based decision support tools has the potential to improve the speed and accuracy of the interpretation of CT imaging in the context of acute stroke.5 6

Brainomix (Oxford, UK) is one of the several commercial enterprises providing a cloud-based software suite, e-Stroke, for automated interpretation of non-contrast CT (NCCT) of the head (e-ASPECTS), intracranial CT angiography (e-CTA) and CT perfusion imaging (e-CTP). e-ASPECTS identifies acute ischaemia in the Middle Cerebral Artery (MCA) territory, anterior circulation vascular hyperdensity caused by acute thrombus, and acute haemorrhage. e-CTA identifies proximal anterior circulation large vessel occlusions. e-CTP automatically post-processes CTP data to provide estimates of the volume of core ischaemia (irreversibly injured brain) and penumbra (hypoperfused tissue at risk that could be salvaged by recanalisation therapy).

Rigorous independent external validation of decision support tools is crucial.7 8 Prior studies assessing the performance of Brainomix e-Stroke, as well as similar software from other vendors, have relied on highly selected cohorts. For example, cohorts often only include patients with confirmed acute ischaemic stroke6 9 and have excluded patients with prior stroke10 or imaging degraded by artefact.11 12 Such study designs increase the risk of bias and have the potential to skew diagnostic statistics.

While Brainomix e-Stroke is designed for use in the setting of acute ischaemic stroke, at our centre and in others, the software automatically analyses imaging from a wider range of patients—patients who will be diagnosed with acute stroke and those who will be diagnosed with a stroke mimic. Therefore, validation in a ‘real-world’ consecutive and unselected cohort is important.

The aim of this study was to provide a ‘real-world’ prospective evaluation of Brainomix e-Stroke software in consecutive, unselected, patients with suspected acute ischaemic stroke admitted to a tertiary centre acute stroke unit.

MethodsCohort

The prospective evaluation of Brainomix e-Stroke was based on a consecutive cohort of patients admitted to the University College London Hospital Hyperacute Stroke Unit (HASU) for suspected acute stroke between 1 October 2021 and 5 April 2022. No patients were excluded based on clinical factors. All patients admitted with suspected acute ischaemic stroke undergo an NCCT of the head and CT angiography (CTA) of the carotid and intracranial arteries on arrival unless contraindicated. CT perfusion (CTP) is used for patients who are eligible for extended time window stroke reversal treatments and with a delayed presentation (>6 hours and <24 hours) or an unknown time of symptom onset.

Imaging data was collected from Carestream PACS (V.12, Philips Medical Systems, Netherlands). Brainomix e-Stroke results were collected from the DICOM-based output available on PACS. Clinical data was collected from a prospectively maintained database of patients admitted to the University College London Hospital HASU.

Patients were excluded if an NCCT was not available or if there were mass lesions diagnosed on the initial CT (eg, metastasis). Patients with a mass lesion that was not identified on the admission CT (but instead was only identified on follow-up imaging) were not excluded. Patients diagnosed with other conditions that did not affect the brain parenchyma (eg, pachymeningeal disease) were not excluded.

CT scanning parameters

Imaging was performed on Siemens X.cite CT scanners. NCCT images were acquired from the skull base to the vertex with the following parameters: slice thickness of 1 mm, 250 mAs, 120 kV and the Hr40 reconstruction kernel. CTA images were acquired from the arch of the aorta to the vertex with following parameters: slice thickness of 0.8 mm, 105 mAs, 90 kVp and the Bv44 reconstruction kernel. CTP images were acquired for a volume extending 9.5 cm from the skull base with the following parameters: slice thickness 5 mm, 170 mAs, 70 kV and the Hr36 reconstruction kernel. 50 mL of Omnipaque 350 iodinated contrast agent was injected at 4 mL/s for CTA and 5 mL/s for CTP.

Brainomix automated analysis

All studies labelled as ‘CT Acute Stroke’ are automatically sent to the Brainomix cloud-based servers for analysis (V.11.1). The output of e-ASPECTS, e-CTA and e-CTP is automatically sent back to PACS in DICOM format. The software was not provided information on the laterality of the symptoms or any other clinical information.

Imaging evaluation

The goal of the study was to determine the true performance of Brainomix e-Stroke (rather than a comparison with a neuroradiologist’s assessment). Therefore, the ground truth for the location of MCA territory acute infarction, hyperdense vessels, acute haemorrhage and large (Internal Carotid Artery [ICA] and/or M1 MCA) and medium (M2 MCA) vessel occlusions was determined using all available imaging (both prior and subsequent) and clinical information by a neuroradiologist. Any uncertainty in the assessment of the images was resolved by consensus with at least two neuroradiologists.

The Alberta Stroke Program Early CT Score (ASPECTS) is a tool for quantifying the size of an acute MCA territory infarct on CT. From a maximum of 10 (ie, normal), the score decreases by 1 point for each affected region where there is parenchymal hypoattenuation and/or swelling.13 For identification of an acute infarct, a true positive was defined as an e-ASPECTS <10 involving at least one correct region on the correct side. In cases where there was bilateral pathology, classification was performed as described previously.14 If e-Stroke identified an acute infarction, vessel occlusion or hyperdense vessel on both sides where there was only a unilateral lesion, this was recorded as a true positive (ignoring the contralateral false positive). Lesions identified by e-Stroke on the wrong side in cases of unilateral disease were recorded as a false negative (ignoring the contralateral false positive).

Acute infarcts were not considered if there was an acute haemorrhage because, by design, information on possible ischaemia is suppressed by e-ASPECTS if a hyperdensity volume (ie, acute haemorrhage) of more than 4 mL is detected.

If there was no follow-up imaging, an ASPECTS of 10 was recorded if an acute infarct was not identified on the initial CT study and if a clinical diagnosis of stroke was not made. Acute infarcts outside the MCA territory were recorded as an ASPECTS of 10. A hyperdense vessel identified by e-ASPECTS was considered a false positive if there was no acute occlusion on the contemporaneous CTA.

Diagnostic statistics for e-ASPECTS were calculated for subgroups based on time (time from symptoms onset/last known well to time of scan of <4.5 hours or ≥4.5 hours) and stroke severity (National Institute of Health Stroke Score (NIHSS) <6 or ≥6).

Analysis of CTP

e-CTP was compared with the output of the perfusion module in syngo.via (V.8.6, Siemens Healthcare GmbH, Germany). The core infarct for e-CTP was defined as the volume of tissue with a Relative Cerebral Blood Volume (rCBV) <30% of the contralateral side. As described previously, a threshold of <20% was used to define core ischaemia in syngo.via.15 Ischaemic penumbra was defined as the volume of tissue with Tmax of greater than 6 s and not already included in the core infarct. The mismatch ratio was defined as the area with a Tmax of greater than 6 s divided by the area of the core infarct. CTP metrics defining eligibility for mechanical thrombectomy were a core infarct volume of less than 70 mL, a mismatch volume of greater than 15 mL and a mismatch ratio of greater than 1.8.16

Statistical analysis

Data is presented as median (IQR) unless otherwise stated. Linear correlation between variables was assessed using Pearson’s correlation coefficient. The Dixon Q-test was used to test for outliers in the accuracy of individual regions for e-ASPECTS. Statistics were performed using statsmodel library as implemented in Python V.3.11.

Brainomix e-Stroke software was made available to our institution through an AI in Health and Care Award, organised by the Accelerate Access Collaborative (AAC) in partnership with NHSx and the National Institute for Health and Care Research (NIHR) in the UK. This service evaluation of the performance of the deployed software was approved as part of the hospital’s clinical governance programme (approval no. 052 02 223-SE).

Results

A flow chart for the study cohort is shown in figure 1. Of the 582 patients who underwent imaging for suspected acute stroke, 11 were excluded because an NCCT was not available (n=6) or because of a diagnosis other than stroke (metastasis in four patients and high-grade glioma in one case) was diagnosed on the original CT. Demographics and clinical information of the cohort are shown in table 1.

Figure 1

Flow chart of the study cohort. †No e-CTA due to no CTA in 13 cases. CTA, CT angiography; CTP, CT perfusion; NCCT, non-contrast CT.

Table 1

Demographics and clinical data of patient cohort

Post-processing success rate and time

In 551 patients admitted during the study period, 1163 studies were performed. Of these, 1130 (97.2%) were successfully processed by e-Stroke. e-ASPECTS produced an output for 551 out of 571 patients (96.5%). While exact reasons for post-processing failure were not provided, there was significant movement artefact in 14 patients. e-CTA successfully provided an output for 545 out of 558 patients (97.7%); in 7 patients there was significant movement artefact and in 3 patients there was poor arterial contrast opacification. e-CTP was successfully post-processed in all patients.

The time from image acquisition to output appearing on PACS was 3 (2–3) min for e-ASPECTS, 7 (5–8) min for e-CTA and 8 (7–10) min for e-CTP. For all studies, the post-processing time was 4 (3–6) min.

e-ASPECTS

Of the 492 patients who did not have a haemorrhage, 128 (26.0%) patients had an acute infarct in the MCA territory.

Figure 2A shows a heatmap of e-ASPECTS and ASPECTS. The median ASPECTS and e-ASPECTS for patients with an acute infarct were 8 (6–9) and 9 (7–10), respectively. e-ASPECTS and ASPECTS were strongly positively correlated within one another (ρ=0.802, p<0.001).

Figure 2

Performance of e-ASPECTS. A heatmap shows strong positive correlation between the ASPECTS and e-ASPECTS (A). Diagnostic statistics are shown for the detection of acute infarct (ie, ASPECTS<10) (B), hyperdense vessels (HD) (C) and acute haemorrhage (D). NPV, negative predictive value; PPV, positive predictive value.

The accuracy for e-ASPECTS in detecting acute ischaemia was 77.0% with a higher specificity (83.5%) than sensitivity (58.6%) and a higher negative predictive value (NPV, 85.2%) than positive predictive value (PPV, 55.6%) (figure 2B). e-ASPECTS and ASPECTS matched in 332 (67.5%) patients and were within 1 point of each other in 429 (87.2%) patients. In patients with an acute infarct, ASPECTS and e-ASPECTS were identical in 28 (21.9%).

Taking an ASPECTS of <6 as a cut-off for mechanical thrombectomy eligibility, there were three patients (0.6% of all patients with ASPECTS ≥6) where a reliance on e-ASPECTS alone would have resulted in an inappropriate exclusion for mechanical thrombectomy.

Figure 3 shows the accuracy by ASPECTS region. Accuracy was lowest in the lentiform nucleus and highest in the M4 region. There were no statistically significant outliers.

Figure 3

e-ASPECTS accuracy by region at the level of the basal ganglia (A) and the supraganglionic level (B).

Of 67 hyperdense vessels in the ICA or M1 MCA, 36 were correctly identified by e-ASPECTS (figure 2C). An accuracy of 69.1% was mainly a consequence of many false positives (n=121).

In some instances, e-ASPECTS (and e-CTA) identified true areas of acute ischaemia (or vessel occlusion) but with insufficient confidence to register as an ASPECTS <10 or a vessel occlusion (online supplemental figure 1).

The diagnostic performance was not significantly different for patients who were scanned before or after 4.5 hours from symptom onset (accuracy of 78% and 76%, respectively, online supplemental figure 3A–D). While there was no significant difference for patients with an NIHSS ≥6, patients with an NIHSS <6 had a significantly lower sensitivity and PPV (21% and 23%, respectively). All 59 patients with acute haemorrhage were identified by e-ASPECTS (figure 2D). Twelve false positives were caused by areas of calcification, typically within the choroid plexus (online supplemental figure 2A).

e-CTA

Of 545 patients with a CTA, there were 80 (14%) vessel occlusions, 62 of which (11% of whole cohort, 77.5% of all vessel occlusions) involved the ICA or M1 MCA. The diagnostic statistics for e-CTA are shown in figure 4. When considering M1 MCA and ICA occlusions, false positives were more common (PPV of 63.4%) than false negatives (NPV of 96.7%). False positives were caused by arterial stenoses and areas of old infarction (online supplemental figure 2B–D).

Figure 4

Performance of e-CTA. Diagnostic statistics for all vessel occlusions (Internal Carotid Artery [ICA], M1 and M2 Middle Cerebral Artery [MCA]) (A) and for large vessel occlusions (LVOs) only (ICA and M1 MCA) (B). CTA, CT angiography; NPV, negative predictive value; PPV, positive predictive value.

e-CT perfusion

Core ischaemia volumes, penumbral tissue-at-risk volumes and the mismatch ratios derived from e-CTP and syngo.via were strongly positively correlated with one another (ρ=0.979, 0.775 and 0.811, respectively) (figure 5).

Figure 5

Scatter plots of the core infarct volume (A), penumbra volume (B) and mismatch ratio (C) volumes derived from Brainomix e-CTP and Siemens syngo.via. Orange data points refer to values greater than 10. CTP, CT perfusion.

In patients with a large vessel occlusion who underwent CTP, eligibility for thrombectomy based on perfusion metrics derived from each software package was concordant in 14 (87.5%) out of 16 patients. In the two discordant cases, the values straddled the eligibility cut-offs (mismatch volume of 14 mL vs 18 mL and core ischaemia 69 mL vs 71 mL).

Discussion

This is the first prospective evaluation of the full suite of Brainomix e-Stroke software in an unselected cohort of patients with suspected acute ischaemic stroke. This study design has the advantage of providing a reliable estimate of both the real-world diagnostic performance (including diagnostic statistics that are influenced by disease incidence) and the post-processing success rates (as all imaging was acquired in line with Brainomix recommendations).

The performance of e-ASPECTS in this study (accuracy, sensitivity and specificity of 77%, 57% and 84%, respectively) falls within the range reported in the literature (67–87%, 14–83% and 57–99%, respectively).11 17 18 Direct comparison with these statistics is difficult because of the significant differences in the study cohorts, which exclusively included patients with a confirmed anterior circulation acute ischaemic stroke. A more direct comparison can be made with a study of e-ASPECTS by Mair et al that included an analysis of a representative cohort that was simulated by enriching CT scans from stroke trial data with scans that were normal or with different diseases. Further, as in this study, the ground truth was based on expert review with all imaging and clinical data. In that analysis, the accuracy, sensitivity and specificity were 71%, 68% and 74%, respectively. Differences in the diagnostic performance may be related to differences in the incidence of stroke in the cohorts (54.4% vs 26.0% in this study) and evaluation of different e-Stroke versions (9 or 10 vs 11 in this study).

There was no statistically significant difference in the accuracy of e-ASPECTS by individual ASPECTS region although the lower accuracy in the lentiform and insular cortex has been reported previously.12

We found e-ASPECTS to be a specific and reliable method of identifying patients for mechanical thrombectomy; only 3 (0.6%) patients were misclassified as ineligible for mechanical thrombectomy based on an ASPECTS of <6. This rate of misclassification is lower than what has been reported previously (3.4%19 and 4.4%14). This misclassification must also be considered in the context of the expanding eligibility criteria for mechanical thrombectomy;20 using an ASPECTS of 3 as a threshold for eligibility, a misclassification by e-ASPECTS would become even less likely. In contrast, algorithm performance was less reliable for patients with an NIHSS of <6, where the sensitivity and PPV were significantly lower than in the whole cohort. This is likely to partly reflect the smaller infarct volume expected in patients with a lower NIHSS.

Identifying hyperdense vessels due to acute thrombus was the least accurate (69.1%) component of e-ASPECTS, which was hampered by a large number of false positives (121, 22.2%). Perhaps higher sensitivity (at the expense of lower sensitivity) is preferable in cases where a patient has presented early and parenchymal ischaemic change has not yet developed; after expert review, a CT angiogram could be performed to exclude acute thrombus. Nevertheless, one must consider the risk of a large number of false-positive hyperdense vessels prompting unnecessary CT angiograms.

e-ASPECTS was highly accurate in detecting acute haemorrhage, which reflects prior studies.21 Accuracy was limited only by false positives mainly caused by choroid plexus calcification. While high sensitivity to acute haemorrhage is important prior to initiating time-sensitive thrombolysis, false positives carry the risk of causing an unnecessary delay while a specialist review is sought.

e-CTA was more specific (93.6%) than it was sensitive (77.6%). Both of these values are higher than a prior study based on the retrospective analysis of 545 CTAs from trial data (sensitivity and specificity of 72%).21 Again, some of this difference may be related to the incidence of large vessel occlusions (53.5% vs 12.5% in this study) and different software versions. Medium vessel occlusions are outside the specifications of the software and therefore, as would be expected, performance for medium vessel occlusions is significantly lower than for large vessel occlusions. Including medium vessel occlusions in the analysis nearly doubled the number of false negatives (15 vs 28). The sensitivity of 65% for large and medium vessel occlusions, which are increasingly considered a target for mechanical thrombectomy, highlights the on-going requirement for specialist review of the CT angiogram (and the requirement for on-going software development).

The values derived from Brainomix e-CTP and syngo.via were positively correlated. The positive correlation was similar to that reported in a study comparing outputs from RAPID AI and Brainomix.15 In contrast to the other components of e-Stroke, the validation of e-CTP is limited by the lack of a readily available ground truth. While comparison of the core ischaemia with contemporaneous diffusion-weighted imaging is possible, this is rarely employed at our centre. Alternatively, the core infarct could be compared with the final infarct volume in patients where complete recanalisation was achieved soon after the CTP. There were too few patients who met these criteria in this study cohort for this to be an option.

A balance must be met between suppressing an output from e-Stroke due to artefact and providing a dependable output. The processing success rate for e-ASPECTS (96.5%) and e-CTA (97.7%) is higher than reported previously (61—89.5%).14 21 22 This likely reflects the homogeneous imaging acquisition at our centre, which has been optimised for use with Brainomix e-Stroke. In contrast, higher failure rates may have been anticipated in prior studies where imaging data was acquired from historical multi-centre trials, often with suboptimal image quality and slice thickness.

The findings from this study are relevant to clinical practice in the UK where investment in AI-based imaging analysis systems is being encouraged at a national level.23 Particularly in centres without immediate 24/7 neuroradiology support, the short processing time and automated analysis of e-Stroke can aid in the early identification of patients who may be candidates for mechanical thrombectomy. Theoretically, this would reduce the time to referral and transfer (ie, reduced door-in-door-out time) to a comprehensive stroke centre for a mechanical thrombectomy, which in turn would improve clinical outcomes. However, our data show that e-Stroke—at the current level of performance—cannot be used in isolation to select cases for active management on a thrombectomy pathway. Clinicians should ensure that all imaging for patients who are eligible for time-critical treatment is still formally reviewed by a radiologist and that cases are not ‘stood down’ for thrombectomy treatment consideration based purely on the results of an automated imaging analysis.

This study has limitations that should be considered. First, we did not compare the performance of Brainomix with a blinded neuroradiologist, which has been reported previously. Rather, the goal of this study was to assess as accurately as possible the true performance of Brainomix in a real-world setting, which requires a firmer assessment of the ground truth. As discussed above, the evaluation of CTP is limited by the lack of a clear ground truth; future studies would ideally compare CTP values with those derived from ‘hyperacute’ MRI or the final infarct volume in those where complete recanalisation was achieved. Our results are based on a tertiary hyperacute stroke centre, where patients routinely undergo video triage before transfer; performance of an algorithm may vary in other centres where the incidence of stroke is lower. A major component of Brainomix e-Stroke is the ability to communicate and transfer images between centres; the value of this functionality was not assessed in this study. Lastly, while the performance of Brainomix software is similar to that reported for other software products,24 direct comparison of Brainomix e-Stroke with other competing products would ideally be performed. Such a comparison, using identical cohorts, would offer valueable information to centres deciding which clinical decision support tool to implement.

Conclusion

e-Stroke is a fast and reliable software package that can analyse CT imaging in the acute stroke setting. However, with wide variation in the accuracy of the individual components of e-Stroke, it remains an adjunct in the interpretation of acute stroke CT imaging and cannot be solely relied on for clinical decision making.