Evaluating the performance and potential bias of predictive models for the detection of transthyretin cardiac amyloidosis

Abstract

Background: Delays in the diagnosis of transthyretin amyloid cardiomyopathy (ATTR-CM) contribute to the significant morbidity of the condition, especially in the era of novel disease-modifying therapies. Screening for ATTR-CM with AI and other algorithms may improve timely diagnosis, but these algorithms have not been directly compared with each other. Methods: We identified patients treated at an integrated health system from 2010-2022 with biopsy or PYP scan-confirmed ATTR-CM and age -and sex-matched them to controls with HF in a 19:1 ratio to target 5% prevalence. We compared the performance of three publicly available algorithms: a random forest model of claims data, the regression-based Mayo ATTR-CM Score, and a deep-learning echo model (EchoNet-LVH). Bias was measured in the best performing models using standard fairness metrics. Results: We identified 198 confirmed cases of ATTR-CM from 3924 patients in the analytic cohort that had the necessary structured and imaging data for all three models. In this cohort, 78.9% of the cohort self-identified as White, 8.7% Black, 4.0% Hispanic and 8.4% Other. ATTR-CM prevalence was highest in individuals who identified as Black. The claims-based model performed poorly with an AUC of 0.48. EchoNet-LVH had higher AUC (0.88 vs 0.78, DeLong Test p < 0.0001) and average precision (0.61 vs 0.15) compared to the Mayo score. Bias auditing of the top two performing models demonstrated that both models satisfied our fairness criteria for equal opportunity (1.05 for EchoNet-LVH and 0.91 for ATTR-CM Score) among patients who identified as Black. Conclusions: In external validation using a large, diverse cohort of patients with heart failure, a deep-learning echo-based model to detect ATTR-CM demonstrated best overall performance compared to two other publicly available models. The results of a bias audit suggest that the regression- and echo-based models are unlikely to exacerbate existing health disparities through inequitable distribution of error with respect to self-identified Black race.

Competing Interest Statement

FSA has received research support from Pfizer and Atman Health. The remaining author report no relevant disclosures.

Funding Statement

This work was support by the American Heart Association (AHA number 856917).

Author Declarations

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

Yes

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

The Northwestern University Institutional Review Board gave ethical approval this study.

I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.

Yes

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

Yes

I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.

Yes

Data Availability

All data produced in the present study are available upon reasonable request to the authors and under the execution of a data use agreement.

留言 (0)

沒有登入
gif