As early as 1979, authors evoked the promise of Artificial Intelligence (AI) to provide diagnostic and therapeutic recommendations for patients in the intensive care unit (ICU) [1]. In the past several years the potential applications of AI in the ICU have skyrocketed [2,3]. Yet, there are few outcome studies of AI models implemented in routine clinical care [2]. This is a notable problem given that AI models are beginning to be integrated into clinical care. Sepsis predictive models for example are currently in use at 54% of large hospitals in the United States [4]. In the midst of this growing use of AI in critical care medicine, it is now more important than ever to ensure that we are addressing sources of bias to promote both fairness and health equity.

AI in health care should support the goal of achieving health equity. The U.S. Department of Health and Human Services, through Healthy People 2030, has defined health equity as “the attainment of the highest level of health for all people” and goes on to say “[a]chieving health equity requires valuing everyone equally with focused and ongoing societal efforts to address avoidable inequalities, historical and contemporary injustices, and the elimination of health and health care disparities [5].” Some have conceptualized bias in AI by using fairness [6,7]. Fairness is “absence of any prejudice or favoritism toward an individual or group based on their inherent or acquired characteristics” [6]. An unfair algorithm is biased toward or against a particular individual or subpopulation [7]. However, fairness is only one approach to achieving an overall goal of health equity. In fact it has been argued that some have pursued mathematical definitions of algorithmic fairness at the expense of substantive equality [8].

Without careful attention to bias across the AI lifecycle, from conceptualization to implementation, there is potential to entrench and exacerbate systemic health care inequities. For critically ill patients, this could manifest as delayed diagnosis or treatment, unequal resource allocation, and poor outcomes (e.g., longer ICU stays, increased complications, worse functional status at discharge, and increased mortality). Critically ill patients are in highly monitored settings, generating substantial amounts of data. Clinicians monitor these data, looking for patterns to guide clinical care. AI models could flourish in this data rich environment, potentially identifying patterns not recognizable to humans. However, with this potential comes the risk of introducing concealed bias.

In this review, we explore sources of potential bias throughout the AI lifecycle, including task definition, data selection, model development and validation, model deployment, model clinical evaluation and continuous model monitoring. We prioritize examples from the critical care literature where possible; however, due to the limited number of studies on AI models implemented in critical care, and even fewer that investigate the equity of these AI interventions, we also incorporate examples from the wider medical literature as well as hypothetical scenarios. We believe this retains value because anticipating bias is difficult, but even hypothetical examples can concretize otherwise abstract concepts. A conceptual framework for bias in the AI lifecycle is shown in Fig. 1.