Cardiovascular disease (CVD) causes more morbidity and mortality than any other clinical condition among citizens of industrialized nations.1 In the US alone, heart disease accounted for 928,741 deaths in 2020 with costs of $407.3 billion ($251.4 billion direct cost and $155.9 billion in lost productivity/mortality).1 Accordingly, significant interest exists to promptly diagnose and triage patients with symptoms of possible coronary syndromes. Various options exist among which clinicians may choose, including no testing, anatomic evaluations such as catheterization or computed tomography, and functional testing by resting and/or exercise or pharmacologic stress testing combined with nuclear, echo, CT, or MRI imaging. Because no tests have perfect accuracy, current guidelines2 advise to interpret cardiac imaging evaluations in the clinical context using Bayesian reasoning of pre- and post-test probability for coronary artery disease (CAD) using some clinical assessment of risk evaluation.

However, estimation of clinical suspicion for CAD remains an imperfect tool.3 One of the core tenets of epidemiology is to be wary of the ecological fallacy—the application of population health estimations to individual risk, particularly for individuals who may differ substantially from the original validation cohort. Classical evaluation of angina typicality based upon Diamond and Forrester (DF) score,4 initially validated among higher risk patients referred decades prior for invasive angiography, leads to over-estimation of disease prevalence in modern patient cohorts with less frequent extent and severity of CAD. Further contributing to over-estimation of classical risk scores is the lower barrier to entry for contemporary testing and associated exponential growth in both appropriate and inappropriate imaging test use.

Several groups have demonstrated the loss of calibration for older risk scores among contemporary cohorts as exemplified by Marcio Bittencourt et al., who showed using coronary computed tomography angiography data from Mass General Brigham healthcare system that the DF pretest risk score categorized only 8.3% of referred symptomatic individuals as requiring no additional testing as compared to 24.6% and 30.0% (P < .001) using two different ESC endorsed CAD consortium clinical risk models. Similarly, DF rated 18% of patients as high pretest probability for obstructive CAD vs 1.1% with the CAD consortium scores (P < .001).3 Along these lines, Rozanski et al. have reported from Cedars Sinai a decreasing prevalence of ischemia diagnosed by SPECT MPI from 29.6% in 1991 to 5.0% in 2009 (P < .001).5 Although, in part, this relates to a shifting epidemiology with widespread adoption of preventive medications such as statins, this also results from exponential growth in availability and use of diagnostic testing in the past few decades. The early 2000s witnessed tremendous growth in cardiac imaging utilization, which led to policy implementation to reinforce the highest yield testing to minimize low-yield care. These policy changes in addition to shifts in economic models led to a leveling off of the growth of cardiac imaging.6 An abrupt curtailment in the use of cardiac imaging occurred during the covid pandemic as noted by Dr. Einstein et al.’ INCAPS survey.7 Despite reductions in low-yield testing, many patients referred to testing have no future adverse cardiovascular event in the next few years, leading to concern regarding how to more precisely triage patients both before and after cardiac imaging. Thus, significant clinical interest exists to evaluate whether mathematical modeling tools may have incremental prognostic value vs current practice of guideline-based clinical Gestalt when combined with cardiac imaging results.

For all these reasons, clinical CAD risk estimators overestimate disease burden. This is particularly true when applying a US-derived risk calculator in Japan, a low CVD prevalence population. Thus, in 2006, began the Japanese Assessment of Cardiac Events and Survival Study by Quantitative Gated SPECT (J-ACCESS) prospective cohort study.8 The original study consisted of 4629 registered patients comprised of 2989 males of mean age 64.9 and 1640 females of mean age 67.2. Now spanning over fifteen years and having published over twenty manuscripts, the J-ACCESS cohort has contributed important findings of CAD prevalence and prognosis for Japanese patients referred for nuclear stress testing. J-ACCESS investigators have furthermore developed a risk table9 validated internally.

In the present manuscript,10 Drs. Petretta and Megna et al. sought to externally validate the J-ACCESS risk estimation with an Italian cohort of 3623 patients who underwent nuclear stress testing between January 2001 and December 2019. Their major finding is that the J-ACCESS model underestimated risk of cardiac death, nonfatal myocardial infarction, and severe heart failure requiring hospitalization among Italian patients. The rate of major adverse cardiac events in the present cohort was twice that in J-ACCESS. Model recalibration resulted in trivial numeric difference in C-statistic increase from .664 to .666 and Brier score decrease from .075 to .073. Overall, the authors demonstrated that use of J-ACCESS did not provide highly accurate risk estimation for the cohort and attempts at recalibration were essentially unhelpful. The J-ACCESS risk estimation when applied to an Italian cohort did not add incremental value to conventional risk estimation, using angina, dyspnea, gender, hyperlipidemia, hypertension, and smoking.

Strengths of the manuscript include the use of a peer-reviewed and internally validated risk estimation tool and vigorous statistical evaluation for external validation. Limitations include significant differences in population including genetic risk differences as Japanese subjects have among the lowest cardiovascular event rates. The Italian cohort also differed with regard to measured risk factors of body mass index, prior coronary artery revascularization, and hypertension. Furthermore, the present study evaluated for heart failure admission, which was not a component of the J-ACCESS outcomes, and would raise the adverse event rate in the Italian cohort.

The valued effort of the authors’ present study serves as a cautionary reminder to procedere con prudenza when applying risk estimators to distinct patient groups and differing clinical endpoints. Several additional notable lessons derive from the interesting paper by Drs. Petretta, Megna et al. First, this paper and others exemplify the exponential growth in recent interest in modeling risk from large datasets using regression techniques, machine learning, and other modeling techniques to explore data within health systems. As noted by a pubmed search (Figure 1) of “risk score registry,” the application of mathematical models to large healthcare datasets ever increases. Such data have the power to transform how we diagnose, prognosticate, and apply therapeutic medical interventions. Never in human history has such a transformational reshaping and growth of scientific knowledge occurred at such an explosive pace that continues to accelerate.11

Manuscripts indexed in PubMed per year containing the keywords “risk score” and “registry”

Second, concurrent to increasing use of large clinical datasets to evaluate for trends, one must not neglect foundational epidemiologic tenets such as the ecological fallacy—be wary of the application of a risk model validated in one population to unique individuals, and moreover to a cohort with disparate baseline characteristics and disease prevalence. One would reasonably anticipate a risk estimator validated in Japan to lose accuracy and reliability when applied in Italy, particularly with the addition of heart failure admission to the combined endpoint.

Third and most importantly, the authors important efforts at external validation underscore the urgent need for cardiovascular research to include more diverse subjects. While the majority of past and present research focuses on white males in Europe and the USA, the current study eloquently demonstrates how demographic differences may significantly affect research outcomes. For these reasons, the US Food and Drug Administration and National Institutes of Health as well as American College of Cardiology, American Heart Association, and European Society of Cardiology and every major scientific body have publicized the urgent need for improved diversity, equity, and inclusion (DEI) in cardiovascular research.12 Steps to improve DEI may invoke health technology including mathematical modeling among disparate cohorts as in the present study as well as machine learning or telemedicine.13 As the authors noted, the J-ACCESS score validated among Japanese patients did not perform particularly well when applied to a dataset derived from an Italian registry. In order to ensure just access to medical care for all, research studies must diversify according to racial, gender, economic, and other demographic variables.

Overall, Drs. Petretta, Megna et al. have contributed an important study10 to validate our understanding of the importance to externally validate risk models. The J-ACCESS registry has yielded insightful findings regarding cardiovascular disease among Japanese subjects. However, one must procedure con prudenza to apply a Japanese cardiovascular risk model in Italy.