The rapid evolution of clinical artificial intelligence (AI) has been spurred by the increasing scope and scale of digital patient data and the emergence and adaptation of specialized machine learning (ML) algorithms. Among others, the field of nephrology is poised to become a prime benefactor of the medical AI revolution, with several retrospective studies highlighting the potential for augmented clinical decision support through data-driven phenotyping and earlier and more accurate prediction of AKI, CKD, renal cell carcinoma, and kidney allograft failure.1 The rise of patient-level ML risk models from retrospective electronic health record (EHR) data suggests the opportunity for more granular and personalized inpatient care strategies. As state-of-the-art AI approaches begin to enter routine clinical practice, it will be critical for multilevel stakeholders to become versed in core AI concepts and methods for harmonious integration of data-driven predictions with existing practices and standards of care.

AI is an umbrella term encompassing a diverse set of data-driven methodologies, and none have been as impactful in recent years as deep learning, the subfield of AI focused on automatically learning optimal representations of raw data. Deep learning has garnered increased interest in the AI-supported nephrology community through positive results from retrospective studies, such as the continuous, early, and accurate forecasting of AKI risk from multidimensional longitudinal patient data using recurrent neural networks (RNNs), a deep learning model designed to understand sequential or temporal data inputs.2 RNNs iteratively examine a patient's longitudinal data record step-by-step through time, and with enough patients, such models can learn global patterns that are optimal for a given prediction target such as AKI. Because deep learning is guided by the principle of automatic pattern discovery, it has the potential to derive novel patient-specific data representations and digital biomarkers that could lead to more precise and tailored treatment strategies.

One of the most significant AI breakthroughs in recent years has been the development of the deep learning transformer model.3 The transformer is functionally similar to an RNN, but instead of analyzing data elements one at a time in chronological order like the RNN, the transformer develops an understanding of an input data sequence by simultaneously comparing the similarity between every pair of elements, resulting in enhanced computational efficiency and a more intuitive way to model each element in a sequence (e.g., in language, the meaning of a word often depends on its relationship to surrounding words). The process of comparing each element in a sequence with all other elements is known as self-attention and is the driving computational force behind the transformer model architecture.

Originally popularized for natural language processing (NLP) tasks, transformers can also aid nephrologists through personalized predictions of patient risk (e.g., short-term AKI, long-term CKD) from a patient's sequence of clinical data measurements commonly found in digital health records (e.g., laboratory tests, vital signs, medications, medical images, clinical notes, and more). Each distinct measurement in a patient's longitudinal health record can be thought of as a health token (similar to how a sentence comprises a sequence of word tokens), and analyzing tokens with transformers can reveal interesting intertoken relationships (such as correspondence between continuous serum creatinine and patient narrative text) that may enable more complex data-driven understandings of overall kidney health and the discovery of subtle digital biomarkers of reduced kidney function. Using AI to analyze relationships between input patient measurements and outcomes can highlight potentially modifiable risk factors that are most influential for prediction. For example, on the basis of the attention similarity score between serum creatinine and urine output measurements, a transformer could infer that for a patient with nonoliguric AKI, the probability of early kidney recovery is associated with urine output volume.

Such opportunities for enhanced transparency can lead to better clinician understanding of AI-driven predictions and facilitate a higher level of clinical trust in prognostication and clinical decision making.

Transformers offer additional benefits in modeling high-dimensional and sparse EHR data where, because of their flexibility due to self-attention, common technical preprocessing steps such as longitudinal resampling and missing data imputation can both be obviated. Transformers are also more adept at capturing longer-term temporal relationships (such as associating signs of drug-induced nephrotoxicity days after initial exposure), a critical step for accurately modeling a patient's entire health history.

AI is currently experiencing a fundamental paradigm shift in which transformers are replacing many traditional deep learning models and are reshaping the way researchers approach many sequential learning applications. While predominantly known for state-of-the-art results in NLP,4,5 transformers are now beginning to demonstrate superior performance across several clinical applications with longitudinal patient data (e.g., Bidirectional Electronic Health Records Transformers for predicting patient diagnoses6). The successful translation of NLP modeling approaches to longitudinal EHR data is best understood through metaphor: Just as text documents are composed of hierarchical sequences of paragraphs, sentences, words, and characters, longitudinal EHRs can be thought of as analogous compositional and temporal sequences of hospital admissions, outpatient clinic visits, medical and diagnostic treatment codes, vital signs, laboratory tests, and even radiography, pathology, and natural language text.

A critical component to the success of many transformer-based approaches is the symbiotic notion of pretraining and transfer learning. In the NLP domain, transformers have shown a remarkable capability to understand the intricacies of human language by learning to predict subsequent (Generative Pre-Trained Transformer 3) or masked (Bidirectional Encoder Representations from Transformers) words from massive corpora of unlabeled text. These pretrained language models can be fine-tuned and adapted to a wide-ranging diversity of downstream prediction tasks. While currently most notable for NLP applications, we anticipate similar effects in the clinical domain using patient data at unprecedented scale made possible by the rise of private and secure pseudo-data sharing techniques such as federated learning.7

Transformer models have already resulted in remarkable advancements in computer vision and NLP and are beginning to make an effect in retrospective patient-level prediction studies. Above all, clinical support tools using health care data and complex transformer technology (Figure 1) offer nephrologists a solution for automatically and intelligently connecting complementary sources of patient data. Personalized models of kidney health resulting from a data-driven understanding across modalities can be used to provide patient-specific risk scores earlier and more accurately than existing approaches and may be able to identify novel digital biomarkers of impaired kidney function.

Figure 1:

Overview of a multimodal transformer framework for modeling longitudinal patient data. EHR, electronic health record.

Given the potential for harm in a nephrology and broader clinical setting, the emergence of these AI frameworks and their eventual integration into patient-facing tools carry significant societal and ethical implications.8 Similar to other AI technologies, transformers carry the potential to perpetuate structural inequities and bias in our health care system. As the representational capacity and predictive capability of transformer-based neural networks continue to rapidly evolve and make their way into clinical practice, it will be critical for nephrologists and medical AI researchers to share a common vocabulary and clearly set expectations and guiding principles to ensure positive and equitable predictions for patients affected by their decisions.

We are in the midst of a fundamental shift and methodological convergence across the AI continuum, and by raising awareness of fundamental transformer modeling approaches across the nephrology community, we can directly engage clinical stakeholders to influence the evolution of these important emerging AI models.

Disclosures

A. Bihorac, P. Rashidi, T. Ozrazgat-Baslanti, and T. J. Loftus report research funding from National Institute of Health. T. J. Loftus also reports research funding from Thomas H. Maren Junior Investigator Fund. P. Rashidi reports ownership interest in Simour.ai and research funding from NIH/NSF. A. Bihorac reports Method and Apparatus for Pervasive Patient Monitoring, US Patent Application Number 20200161000, filed January 6, 2018; Systems and Methods for Providing an Acuity Score for Critically Ill or Injured Patients, US patent WO2020172607A12020; and Method and Apparatus for Pervasive Patient Monitoring, US Patent Application Number 20190326013, filed April 18, 2019. B. Shickel reports a patent application for Systems and Methods for Providing an Acuity Score for Critically Ill or Injured Patients. A. Bihorac, T.J. Loftus, T. Ozrazgat-Baslanti, P. Rashidi, and B. Shickel report Provisional Appl. No 62/809,159, filed February 22, 2019. All remaining authors have nothing to disclose.

Funding

This work was supported by grants K01DK120784 and R01DK121730 from the National Institute of Diabetes and Digestive and Kidney Diseases, grants K23GM140268 and R01GM110240 from the National Institute of General Medical Sciences, grant R01NS120924 from the National Institute of Neurological Disorders and Stroke, and grant R01EB029699 from the National Institute of Biomedical Imaging and Bioengineering (NIH/NIBIB).

Acknowledgments

This article is part of the Artificial Intelligence and Machine Learning in Nephrology series, led by series editor Girish N. Nadkarni.

The content of this article reflects the personal experience and views of the authors and should not be considered medical advice or recommendation. The content does not reflect the views or opinions of the American Society of Nephrology (ASN) or CJASN. Responsibility for the information and views expressed herein lies entirely with the authors.

Author Contributions

B. Shickel conceptualized the study; A. Bihorac and P. Rashidi provided supervision; B. Shickel wrote the original draft; and A. Bihorac, T.J. Loftus, T. Ozrazgat-Baslanti, Y. Ren, and B. Shickel reviewed and edited the manuscript.

References 1. Loftus TJ, Shickel B, Ozrazgat-Baslanti T, et al. Artificial intelligence-enabled decision support in nephrology. Nat Rev Nephrol. 2022;18(7):452–465. doi:10.1038/s41581-022-00562-3 2. Tomašev N, Glorot X, Rae JW, et al. A clinically applicable approach to continuous prediction of future acute kidney injury. Nature. 2019;572(7767):116–119. doi:10.1038/s41586-019-1390-1 3. Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. Adv Neural Inf Process Syst. 2017;30:5998–6008 4. Devlin J, Chang M, Lee K, Toutanova K. BERT: pre-training of deep bidirectional transformers for language understanding. arXiv. 2018. doi:10.48550/arXiv.1810.04805 5. Brown T, Mann B, Ryder N, et al. Language models are few-shot learners. Adv Neural Inf Process Syst. 2020;33:1877–1901. 6. Li Y, Rao S, Solares JRA, et al. BEHRT: transformer for electronic health records. Sci Rep. 2020;10(1):7155–7212. doi:10.1038/s41598-020-62922-y 7. Rieke N, Hancox J, Li W, et al. The future of digital health with federated learning. NPJ Digit Med. 2020;3(1):119. doi:10.1038/s41746-020-00323-1 8. Bommasani R, Hudson DA, Adeli E, et al. On the opportunities and risks of foundation models. arXiv. 2021.doi:10.48550/arXiv.2108.07258