IntroductionBackground

The growing adoption of electronic medical records (EMRs) in many high-income countries has resulted in improvements in health care delivery through the implementation of clinical decision support systems at the point of care []. To meet the ever-accelerating demands for clinical care, various innovative models have been developed to harness the potential of EMR data [-]. These new care models aim to enable health care organizations to achieve the quadruple aim of care, which includes enhancing patient experience, advancing providers’ experience, improving the health of the population, and reducing health care costs [].

Artificial intelligence (AI) holds the potential to improve health system outcomes by enhancing clinical decision support systems [,]. AI aims to augment human intelligence through complicated and iterative pattern recognition, generally on large data sets that exceed human abilities []. While a large body of academic literature has demonstrated the efficacy of AI models in various health domains, most of these models remain as proof of concept and have never been implemented in real-world workflows []. This demonstrates the relatively inconsequential endeavors of many AI studies that fail to produce any meaningful impact in the real world. Even with the substantial investments made by the health industry, the implementation of AI analytics in complex clinical practice is still at an early stage []. In a limited number of instances, AI has been successfully implemented, largely for nonclinical uses such as service planning or trained on limited static data sets such as chest x-rays or retinal photography []. The factors influencing the success or failure of AI implementations in health are poorly investigated []. Understanding these barriers and enablers increases the likelihood of successful implementation of AI for the digital transformation of the health system [,], ultimately aiding in achieving the quadruple aim of health care [].

Toward the Digital Transformation of Health Care

A 3-horizon framework has been previously published to help health systems create an iterative pathway for successful digital health transformation ( []). Horizon 1 aims to optimize the routine collection of patient data during every interaction with the health system. In horizon 2, the data collected during routine care are leveraged in real or near real time to create analytics. Finally, in horizon 3, the insights from data and digital innovations are collated to develop new models of care. A health care system focused on continuous improvement is referred to as a learning health system (LHS) that uses routinely collected data to monitor and enhance health care outcomes consistently []. When health care organizations reach the third horizon, they can leverage data in near real time to create ongoing learning iterations and enhance patient care, leading to the establishment of an LHS [].

‎Figure 1. The 3-horizon framework for digital health transformation (adapted from Sullivan et al [] with permission from CSIRO Publishing).

Regarding the 3-horizon model, EMRs are the foundation of horizon 1 (). While many health organizations have successfully adopted EMRs into their existing workflows, the transition to horizons 2 and 3 has been challenging for many of these health care facilities []. A critical phase in this transition involves moving beyond the capture of EMR data for delivering analytics, including AI, aiming to improve clinical outcomes. There is little published evidence to assist health systems in making this transition [,].

Analysis of Prior Work

Before conducting our review, we performed a manual search on Google Scholar using our Medical Subject Heading (MeSH) terms along with the “review” keyword to identify previous review papers that aimed at reviewing studies on the implementation of clinical AI in health care settings. We also included review papers known to our research team. Between 2020 and 2022, we identified 4 reviews that were relevant to the implementation of AI in health care systems [-]. Overall, these papers reviewed 189 studies between 2010 and 2022. The characteristics of these reviews, outlined in , were the year of publication, the targeted care settings, the source of data, the predictive algorithm, and whether the predictive algorithm was implemented.

Table 1. The inclusion criteria for this study and previous work.StudyYearHealth care settingData sourcePredictive algorithmImplementation stateLee et al []2020AnyEMRaAnyImplementedWolff et al []2021AnyAnyAIb and MLcImplementedSharma et al []2022AnyAnyAI and MLImplementedChomutare et al []2022AnyAnyAI and MLImplemented or developedOur study2023HospitalsEMRAI and MLImplemented or guidelines

aEMR: electronic medical record.

bAI: artificial intelligence.

cML: machine learning.

The prior works identified 20 enablers and 13 barriers to AI implementation in health care across 4 categories: people, process, information, and technology ( [-]). Overall, the findings derived from these review papers hold significant potential in providing valuable insights for health systems to navigate the path toward digital health transformation. One prevailing shortcoming of these studies is the absence of alignment with evidence-based digital health transformation principles to provide health care organizations with actionable recommendations to enable an LHS [], therefore limiting their applicability for strategic planning within hospital organizations.

Research Significance and Objectives

Hospitals are intricate hubs within the health care ecosystem, playing a central role in providing comprehensive medical care and acting as crucial pillars supporting the foundations of health care systems worldwide. Understanding the factors influencing the success or failure of AI in hospitals provides valuable insights to optimize the integration of these emerging technologies into hospital facilities. While the previous reviews included all health care settings [-], our study only focused on hospital settings. Given the limited instances regarding the implementation of AI in hospital facilities, this study explored the real-world case studies that have practically reported their AI implementation solutions in hospital facilities, aiming to synthesize the evidence of enablers and barriers within their implementation process. In addition to the inclusion of these implementation case studies, we incorporated implementation guidelines as they can potentially assist in the overall understanding of AI implementation in hospitals. This study also focused on aligning the evidence of enablers and barriers within the 3-horizon framework [], offering a way to establish an empirical infrastructure. As a result, this can enable health care organizations to learn, adapt, and accelerate progress toward an LHS [].

This review investigated the following research questions (RQs): (1) What enablers and barriers are identified for the successful implementation of AI with EMR data in hospitals? (RQ 1) and (2) How can the identified enablers of and barriers to AI implementation lead to actions that drive the digital transformation of hospitals? (RQ 2).

In addressing these questions, our objectives were to (1) conduct a systematic review of the literature to identify the evidence of enablers of and barriers to the real-world implementation of AI in hospital settings and (2) map the identified enablers and barriers to a 3-horizon framework to enable the successful digital health transformation of hospitals to achieve an LHS.

MethodsSearch Strategy

This study followed an extended version of the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines to outline the review methodology with comprehensive details []. PubMed, Scopus, Web of Science, and IEEE Xplore were searched on April 13, 2022. We reviewed prior work to determine potential MeSH keywords relevant to our study [-]. A research librarian helped with the definition of the MeSH keywords in PubMed and the translation of that search strategy to all platforms searched. The search strategies were applied across the 4 databases (). The MeSH keywords used to search PubMed were as follows: product lifecycle management, artificial intelligence, machine learning, deep learning, natural language processing, neural networks, computer, deep learning, big data, hospital, inpatient, medical, clinic, deploy, integrate, monitor, post prediction, data drift, and regulatory. Using the Boolean operator OR, their synonyms were joined to form search phrases. Combining search phrases using the AND operator produced the final search string. We incorporated the term “data drift” to the title and abstract, and full-text search as it is a prominent concept for the continuous integration of AI. The term “regulatory” was also added to our search criteria because it is a relevant term for the implementation of AI in health care within the domain of software as a medical device. The reference lists of the included studies were examined to ensure that all relevant papers were included.

Eligibility Criteria

The inclusion criteria were articles published from January 1, 2010, to April 13, 2022, that included case studies and guidelines on the implementation of AI analytic tools in hospital settings using EMR data. Given the scarcity of real-world AI tools in hospital settings, especially the scarcity of published case studies of unsuccessful implementations of clinical AI tools, we specifically included case studies that successfully implemented AI within hospitals to understand lessons learned and provide use cases that other jurisdictions may learn from. On the basis of a review of frameworks for AI implementation in health care practice, we defined the term implementation as “an intentional effort designed to change or adapt or uptake interventions into routines” []. The term “barrier” was defined as “experiences that impeded, slowed, or made implementations difficult in some way” []. In contrast, the term enablers was defined as factors, experiences, or processes that facilitated the implementation process. Studies conducted in community or primary care settings were excluded as our main focus was hospital facilities. Studies that did not use AI models were also excluded. We also eliminated non–English-language and conference articles. Studies that focused on regulatory domains and challenges, opportunities, requirements, and recommendations were also excluded as they did not demonstrate real-world AI implementation. The selection of studies was based on the criteria specified in .

Textbox 1. Inclusion criteria for this study.

Inclusion criteria

Population: adults (aged ≥18 y); inpatientsIntervention: successfully implemented artificial intelligence (AI) and machine learning (ML) tools using hospital electronic medical record dataStudy design: case studies that implemented AI and ML in the real world; guidelines on the real-world implementation of AI and MLPublication date: January 2010 to April 2022Language: English

Exclusion criteria

Population: nonadults (aged <18 y); outpatientsIntervention: traditional statistical methods; rule-based systems; systems without AI and MLStudy design: studies without implementation of AI and ML; studies focused on AI and ML development, regulatory-related domains, challenges, opportunities, and recommendations; conference papers; primary care or community settingsLanguage: non-EnglishScreening

For the screening and data extraction procedures, the Covidence (Veritas Health Innovation) systematic review software was used []. A 2-stage screening process was performed with the involvement of 2 reviewers (AKR and OP). In the initial stage, the reviewers assessed the relevance of titles and abstracts based on the inclusion criteria. Subsequently, in the second stage, the full texts of the included articles were reviewed by AKR and OP independently. Consensus was reached through discussion between the reviewers whenever necessary.

Data Extraction and Synthesis

AKR and OP conducted the procedure of data extraction. The following study characteristics were extracted from all final included studies: country, clinical setting, study type (case study or guideline), and aim of study. With the adoption of EMR as a prerequisite for AI development, our focus was on extracting evidence of enablers and barriers solely within horizons 2 (implementation) and 3 (creating new models of care). In total, 2 reviewers (AKR and OP) independently extracted evidence regarding enablers and barriers (RQ 1), subsequently reaching consensus through weekly discussions and analysis. The extracted data were disseminated among our research team for review and to gather additional feedback.

To address the second RQ (RQ 2), we mapped the findings from previous reviews along with the found factors in this study across horizons 2 and 3 of the digital transformation framework []. Following the data extraction phase, 2 reviewers independently mapped the identified enablers and barriers to 4 categories (people, process, information, and technology). During the mapping of a given enabler or barrier, if it was related to the development of AI analytics, it was mapped to horizon 2 considering its relevance across the 4 domains (people, technology, information, and processes). When an enabler or barrier was associated with the postdevelopment phase focusing on establishing new care models, it was mapped to horizon 3. Consensus was reached between AKR and OP through a meeting to finalize the mapping phase.

Quality Assessment

For the included use case studies, we used the Mixed Methods Appraisal Tool (MMAT) [] to conduct a quality assessment. The choice of the MMAT was suitable as the included use case studies exhibited a range of qualitative, quantitative, and mixed methods designs. For evaluating the methodology of guideline studies, we followed the ADAPTE framework []. With 9 modules for guideline development, this framework was designed to streamline and enhance the process of creating guidelines within the health domain. The quality assessment was conducted independently by 2 authors (AKR and OP), and any discrepancies were resolved through a meeting.

ResultsStudy Selection

The search strategy retrieved 1247 papers from PubMed, Scopus, IEEE Xplore, and Web of Science for analysis, and 67 (5.37%) duplicates were identified and eliminated using the EndNote (Clarivate Analytics) citation manager. After screening titles and abstracts, 92.03% (1086/1180) of the studies were removed as the inclusion criteria were not satisfied. A total of 7.97% (94/1180) of the papers remained for full-text review following title and abstract screening. In total, 48% (45/94) of papers were excluded because AI models were not implemented in clinical care. A total of 19% (18/94) of the studies were excluded because they focused on regulatory domains. In total, 9% (8/94) of the studies were excluded due to being the wrong intervention (eg, studies that did not develop AI models). A total of 3% (3/94) of the studies were found to have a clinical population that did not align with our inclusion criteria (eg, hospitalized patients). One study was not in English and was excluded. In addition, 7 studies were discovered by scanning the reference lists of the included articles. In total, 26 studies were included in this review, comprising 9 (35%) guideline studies and 17 (65%) papers with successful implementation examples (). presents the PRISMA flow diagram outlining the outcomes of this review.

Table 2. Characteristics of the studies included in this review.Study, yearCountryClinical settingStudy typeAim of studyEnablersBarriersWilson et al [], 2021United KingdomGeneralGuidelineTo provide advice from health care experts on clinical AIa development and implementation
A team of multidisciplinary experts, including clinicians, software developers, data scientists, and hospital IT staff
Senior, experienced individuals can be particularly useful to overcome implementation barriers
The appointment of a data champion
Staff training in the data science field
Using data scientists or a trusted research environment with appropriate tools can ensure adequate data privacy
A common language with necessary terminologies is suggested within the CSTb
Clinicians can assist in understanding and resolving the quality and reliability of AI solutions
HCPs’c inexperience with AI
The integration of disparate data sources is one of the barriers to AI solutions in the current workflow
Svedberg et al [], 2022SwedenGeneralGuidelineTo develop an AI implementation framework in health care
To conduct AI implementation studies to provide direction for further improvement of the framework
To implement the proposed framework in routine care
The co-design process among clinicians, data scientists, and end users
The national and regional initiatives to facilitate AI implementation into practice
Several major investments facilitated the establishment of the infrastructure design and development of this study
Literature review and the existing theory-driven frameworks and strategies
Technological knowledge and awareness of challenges, including social, cultural, and organizational barriers
Lack of accessibility of AI implementation science to individuals who could potentially benefit from it
Subbaswamy and Saria [], 2019United StatesGeneralGuidelineTo explain data shift and overview the types of existing solutions
Graphical representation can be used to assess the stability of AI models and identify potential performance shifts but requires domain for interpretation
Proactive learning approaches allow models to be stable against anticipated shifts in the future, including the use of stable algorithms that are robust to future shift
Data set shift is prevalent and problematic in clinical AI settings and needs to be accounted for to prevent performance decay
Pianykh et al [], 2020United StatesRadiologyGuidelineTo examine the key principles and issues involved in integrating AI with continuous learning in radiology
Radiologists and clinicians are important to the successful implementation of continuous-learning AI to provide feedback
Continuous learning is a viable method to combat data drift
Leiner et al [], 2021The NetherlandsRadiologyGuidelineTo demonstrate the necessity for a vendor-neutral AI implementation infrastructure
To provide a plan for a vendor-neutral AI implementation infrastructure
To discuss prominent issues, including governance, quality control, and ethics
A team of multidisciplinary experts, including clinicians, data scientists, and IT staff
Platforms are suggested as vendor-neutral infrastructures shared by researchers and clinicians and allow AI systems to receive iterative feedback from clinicians
The accessibility of the AI results at the time of care without requiring physicians to switch workstations or launch specialized software
Consistency between AI implementation methods used within one hospital
The messaging standards, such as HL7d
Using the containerization concept to concurrently run multiple instances of AI analytics
Training end users and clinicians for using and interpreting AI results
Gruendner et al [], 2019GermanyGeneralGuidelineTo implement a secure platform to develop and deploy MLe models in health care settings
The FHIRf standard was used to exchange health data between different health care points in a consistent manner
The OMOP-CDMg database structure was used as a standard method to organize health care data consistently across various data points. This also enabled the availability of data to researchers and end users.
Containerization allowed for a flexible development environment. It enabled clinicians and ML developers to collaborate and improve performance.
The proposed platform provides scientists with a secure, privacy-preserving, flexible research infrastructure to develop and deploy statistical models within a hospital’s IT infrastructure
Using appropriate data privacy techniques can allow for model training using data from multiple hospitals in parallel
Collaboration among the research team
Predictions can be extremely slow with large input data due to hardware limitations; therefore, the AI may output results not in real time
Generalizable platforms such as KETOS are versatile, but as a result, they are relatively inefficient and may require further customizations and fine-tunings at the local level
Eche et al [], 2021United StatesRadiologyGuidelineTo provide strategies to tackle overfitting and underspecification of AI models
Underspecification (the lack of generalizability) can be addressed with the use of artificial or real shifts in test data
Overfitting and underspecification can negatively impact the generalizability of AI in health care
There is a trade-off between performance and generalizability when addressing underspecification
Allen et al [], 2021United StatesRadiologyGuidelineGuideline of evaluation of AI in a radiology setting before implementation in the workflow to assist in purchase decisions and monitoring of the performance afterward
Enriched site-specific data can facilitate AI evaluation, allowing that the target population is well-represented before implementation
In the AI evaluation process, capturing the metadata about equipment manufacturers, the protocol used, and demographics in the AI data registry can reveal performance decline and show whether the decline is related to specific machines or manufacturers
QAh allows AI to perform according to the implementation requirements
Model evaluation can be difficult and restricted to larger, informatics-familiar institutions
Verma et al [], 2021CanadaGeneralGuidelineTo provide an approach for developing and implementing AI in health care
Multidisciplinary team
Safety monitoring
Data quality
User-friendly user interface
Nondisruptive to the current workflow
End-user trust
Continuous evaluation of performance
Wiggins et al [], 2021United StatesRadiologyCase studyTo develop an AI solution that can generate, consume, and provide outcomes within the clinical radiology process
Collaboration among developers, radiologists, and AI vendors
Interoperability standards and robust methodologies, such as HL7, FHIR, and SOLEi
The use of metadata such as hardware or software specifications
Radiologists should be able to provide feedback on AI results
Raising awareness and providing the required training regarding the potential of AI technologies among clinicians and patients can help increase AI adoption
Wang et al [], 2021ChinaRadiologyCase studyTo create an AI system that analyzes CTj scans automatically to promptly detect COVID-19 pneumonia in hospitals
Co-designing with clinicians
The AI model was externally validated to assess the generalizability before deployment
Preconfigured model development allowed for very quick deployment
Continuously collected data can lead to better generalizability of AI products and are considered a crucial aspect of epidemic response
Lack of reception of continuous data for retraining the model may result in data drift and underfitting
Strohm et al [], 2020The NetherlandsRadiologyCase studyTo explore barriers to and enablers of AI implementation in radiology
Collaboration among HCPs in radiology
Financial challenges in the Dutch health care system
The optimism toward AI potential
The existing strategies and initiatives in digital health
The appointment of a data champion
Inconsistent efficacy of AI output
Lack of robust implementation procedures
Unclear added value of AI applications in routine care
Trust issue of HCPs
Soltan et al [], 2022United KingdomEDk triageCase studyTo implement an AI application to screen patients with COVID-19 in an ED and perform multicenter external validation
Conducted multicenter validation across 4 hospitals, including both temporal and geographical validations
Deployment occurred in parallel with the preexisting method, allowing for a direct comparison of performance
The AI only using laboratory tests already routinely done allowed for minimal interruption of regular clinical workflow
Temporal and geographical external validation allowed for the assessment of the generalizability of the AI tool
Validation only performed in 1 geographical region
Sohn et al [], 2020United StatesRadiologyCase studyTo develop an infrastructure for the implementation of ML models in routine radiology workflow
Collaboration of a multidisciplinary team
The minimum disruption to the current workflow can increase the AI uptake
An open-source pipeline facilitates the integration of additional algorithms
An ML model agnostic to the hospital systems for easier modification and retraining without impacting the existing infrastructure
The use of a QA framework by end users, clinicians, and software testers to identify model errors and submit those errors for model update
Minimum disruption to the existing radiology workflow
QA evaluation
A dedicated server for the AI applications
Preexisting pipelines for clinical AI deployment often rely on third-party software, which can be problematic due to complexity, privacy, and maintenance issues
Pierce et al [], 2021United StatesRadiologyCase studyTo implement an AI-enabled mobile x-ray scanner detecting pneumothoraxes in a radiology clinical workflow
Compatibility of the clinic’s system with the vendor along with the vendor’s willingness to collaborate
Granting user access privileges according to their specific roles
Staff training
The model received continuous training
Training and education of users in the use of AI can be beneficial
Minimum disruption to the current workflow
Kanakaraj et al [], 2022United StatesRadiologyCase studyTo develop and demonstrate a clinical image AI validation tool with a convenient user-friendly front end while meeting important security and privacy standards
Use of secure software—PACSl image management, HTTPS service, and REDCapm database
The AI imaging incubator successfully provided an architecture for executing clinical AI models and displaying results in a clinician-friendly manner while meeting key security and privacy standards (HIPAAn compliance)
Lack of appropriate procedure to capture users’ feedback for continuous improvement of AI model
Jauk et al [], 2020AustriaGeneralCase studyTo implement ML models to forecast the occurrence of delirium among patients admitted to hospitals
Clinical staff were involved in the implementation process
Training for nurses and physicians involved is beneficial
Performance analysis can be complicated for early-warning intervention AI systems
The incidence of delirium was lower than anticipated, impacting the calibration
Sometimes, the algorithm would underperform on patients with fewer previous hospital stays due to reduced EHRo data
Davis et al [], 2019United StatesGeneralCase studyTo outline a procedure for selecting updating methods to combat clinical prediction model drift
The procedure effectively recommended updating methods proportional to the need
This procedure can be applied to any type of model
The procedure is conservative compared with others
The procedure provides no guarantee of clinically appropriate improvement to model performance
Blezek et al [], 2021United StatesRadiologyCase studyTo outline and demonstrate a system for general AI deployment in radiology and discuss use cases and requirements
The Agile development approach was used to deliver the AI product
Radiology IT support was significantly involved
Computational and storage resources were appropriately configured to properly handle the current and future processing requirements
Radiologists received training on the use of the new system
Custom solutions can fit and function seamlessly in clinical workflows but are susceptible to some issues
Radiologists approved of the ability to conveniently decide the correctness of the results and the system’s seamless and intuitive integration into their workflow
Vended implementation platforms are also imperfect
Pantanowitz et al [], 2020United StatesPathologyCase studyTo clinically validate an AI algorithm for detecting prostate adenocarcinoma, grade tumors, and detect clinically important features
To deploy the AI algorithm in clinical workflow
Substantial increase in pathology workload and job complexity makes it a prime candidate for AI uptake
External validation
The use of unseen data sets for performance validation
Small calibration data set was effective for adapting the algorithm to a new environment
Combining target categories into clinically significant groups reduced computational requirements, allowing for real-time analysis
Discrepancy in labeling data due to discordance among physicians for cancer grading
Fujimori et al [], 2022JapanEDCase studyTo evaluate the enablers of and barriers of implementing AI in emergency care
Data explanation and visualization were used to justify the alerts
Robust validations are required to avoid undesired consequences
Alert fatigue was avoided by processing background information and presenting visual data
Training clinicians
Low performance in workflow
Alert fatigue
The risk of bias on a clinician’s decision when using the AI application
Joshi et al [], 2022United StatesGeneralCase studyTo examine the implementation of a sepsis CDSp tool with ML models and rule-based approach from the viewpoint of those leading the implementation
Ease of integration and ability to customize the AI model
Difficulties with the definition of optimal alerts
Alerts were said to be disruptive to the workflow
Alert fatigue
Concerns about the clinical relevance of the new system
Difficult to explain and understand ML outputs
Trust issue with the output due to misunderstanding the output
High financial cost
Pou-Prom et al [], 2022CanadaGeneralCase studyTo develop an AI application for predicting the risk of clinical deterioration in hospitals
Multidisciplinary team
Security measures were adopted
Clinical relevance to the targeted cohort
Temporal validation
Conducted a pilot test to understand the model output
User training
Model update to avoid data drift
Lack of external validation
Lack of generalizability
Baxter et al [], 2020United StatesGeneralCase studyTo identify barriers to AI uptake in workflow
End users’ concerns about whether the new solutions are relevant to their workflow
Potential disruption to the routine workflow and unintended consequences
Lack of customization capability
Sandhu et al [], 2020United StatesEDCase studyTo examine the variables influencing the implementation of ML applications for predicting sepsis incidence
Co-design with nurses and clinical staff
Introduced a new job title responsible for the integration
Having the required clinical knowledge about sepsis
Training end users
Clinicians’ trust
Lack of understanding of the output
Alert fatigue
Disruption to the workflow
Sendak et al [], 2020United StatesEDCase studyTo report a deep learning sepsis detection and management system
Multidisciplinary team
Co-design with clinical staff
Hospital leaders and external research partners
Training staff
Data scientists with the required clinical background
Personnel time for integration of new ML system
Shared infrastructure for development and deployment
Lack of evidence-based implementation guidelines
Disruption to the workflow
Lack of feedback loop for continuous updating

aAI: artificial intelligence.

bCST: collaborative science team.

cHCP: health care provider.

dHL7: Health Level 7.

eML: machine learning.

fFHIR: Fast Healthcare Interoperability Resources.

gOMOP-CDM: Observational Medical Outcomes Partnership Common Data Model.

hQA: quality assurance.

iSOLE: Standardized Operational Log of Events.

jCT: computerized tomography.

kED: emergency department.

lPACS: picture archiving and communication system.

mREDCap: Research Electronic Data Capture.

nHIPAA: Health Insurance Portability and Accountability Act.

oEHR: electronic health record.

pCDS: clinical decision support.

‎Figure 2. PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flowchart for study selection. Study Characteristics

outlines the characteristics of the included studies in this review. The publication dates of the included studies ranged from 2019 to 2022 [,-]. In total, 65% (17/26) of the studies were case studies on the implementation of AI in hospitals [,-], whereas the remaining 35% (9/26) were implementation guidelines [-].

Of the 26 identified studies, 15 (58%) originated from the United States [,,,,,,-,-,-]; 2 (8%) originated from the United Kingdom [,]; 2 (8%) originated from the Netherlands [,]; and 1 (4%) originated from China [], Australia [], Japan [], Canada [], Austria [], Germany [], and Sweden [] each.

Radiology was the clinical setting in 46% (12/26) of the studies [,,,,-,-,,]. A total of 38% (10/26) of the studies were conducted in general inpatient wards [,-,,,,,,], and 15% (4/26) were conducted in emergency departments [,,,].

Quality Assessment

Regarding the 35% (9/26) of guideline studies, none fully adhered to the ADAPTE framework []. Although these included guideline studies had clear scopes and purposes aligned with this review, they all lacked details concerning the assessment of quality, external validation, and aftercare planning procedures. The details of this assessment for all the guideline studies can be found in [,-].

With respect to the 65% (17/26) of case studies, they were classified into 3 groups: quantitative descriptive (12/17, 71%) [,,-,,], qualitative (4/17, 24%) [,,,], and mixed methods (1/17, 6%) []. Overall, 5 of the case studies met the MMAT criteria: all 4 (80%) qualitative studies and the one mixed methods study. The remaining 71% (12/17) of quantitative descriptive studies failed to fully adhere to the MMAT criteria. In all but 17% (2/12) of these quantitative descriptive studies, an appropriate data sampling strategy was not used to represent their target population [,]. The statistical analysis of the findings was assessed as appropriate in 58% (7/12) of the quantitative descriptive studies [,,,,,,]. Overall, our assessment revealed that the quality of 81% (21/26) of the included studies was poor due to insufficient reporting of their methodologies ().

RQ FindingsRQ 1A Findings: Enablers of AI Implementation in Hospitals

A total of 28 enablers extracted from both prior work and this study (n=8, 29% were new enablers identified in our study) are presented in . Most of these newly identified enablers (7/8, 88%) related to the information and technology categories, highlighting the potential opportunities for hospitals regarding data readiness and required technologies for the successful implementation of AI. A total of 54% (15/28) of the enablers were shared findings between the previous reviews and this study.

Table 3. Consolidated view of research question 1A (enablers to artificial intelligence [AI] implementation; N=26)a.

aEnablers identified in previous reviews and this review were mapped to 4 categories of the 3-horizon framework [].

bNot specified.

Within the scope of the 3-horizon framework [], most included studies in this paper (22/26, 85%) indicated that the process domain facilitated the development of AI analytics within horizon 2 [,,,-,,,-]. Co-design with clinicians was the most commonly reported enabler in 46% (12/26) of the papers in horizon 2 [,,,,-,,,-]. The process domain was also highlighted as having a facilitative role in the creation of new care models with AI (horizon 3) in 35% (9/26) of the papers [,,,,,,,,]. Training end users to adopt AI solutions and interpret the insights was reported in all these 9 studies as an enabling factor in horizon 3.

Technological factors were highlighted in 58% (15/26) of the studies as enablers within horizon 2 [,-,,,-,-,], with the most commonly reported factor being continuous learning capability of AI analytics [,,,,] and containerization capability by providing separated development environments [,,,,] and applying the interoperability techniques ensuring seamless integration of diverse formats of clinical data from different hardware and software sources [,,,].

Of all the included studies, 46% (12/26) [,,,-,,,,,] and 31% (8/26) [,,,,,,,] identified people-related enablers across horizons 2 and 3, respectively, with multidisciplinary teams in horizon 2 and trained end users in horizon 3 being the 2 most reported enablers.

Enabling factors related to the information domain were discussed in 35% (9/26) of the included studies in this review [,,,,,,-], with data quality being the most reported enabler of the successful implementation of AI in hospitals in >50% of these papers (5/9, 56%) [,,,,]. The enablers of the AI adoption in hospitals were reported to include factors such as considerations of data security [,,] and data visualization [,] in horizon 2 along with AI usability [] solutions in horizon 3.

RQ 1B Findings: Barriers to AI Implementation in Hospitals

Overall, a total of 18 barriers to AI implementation in hospitals were extracted from both prior work and this study, with 5 (28%) found to be new in this study (). Most of these newly identified barriers (4/5, 80%) were related to the information and technology categories. A total of 50% (9/18) of the identified barriers were found to be shared findings between the previous work and this study. In our analysis, some factors played dual roles, acting as both enablers and barriers. For instance, “Seamless integration” served as an enabler (enabler 5; ), whereas “Disruptive integration” acted as a barrier (barrier 3; ). We reported both enablers and barriers with such reversed meanings to highlight the real-world complexities due to which such factors can exhibit this duality.

Regarding the 3-horizon framework [], 58% (15/26) of the included studies in this review showed that the process domain hindered the development of AI within horizon 2 [,,,-,-,-]. The lack of sufficient performance assessment within horizon 2 was the most commonly reported barrier in 27% (7/26) of the papers [,,,,,]. The factors related to the process domain were also reported as barriers to the implementation of AI within horizon 3, with 8% (2/26) of the papers reporting alert fatigue as an obstacle to AI adoption for creating new models of care [,].

Information-related factors were highlighted in 31% (8/26) of the studies as barriers within horizon 2 [,,,,], with the most commonly mentioned one being poor data quality [,,,,]. The challenge with data shift was reported as part of the information domain within horizon 3 [].

Technology-related challenges in horizon 2 were identified in 19% (5/26) of the studies, including issues such as the lack of customization capability and computational limitations of hardware [,,,,].

Within horizon 3, a total of 19% (5/26) of the included papers highlighted the barriers related to the people domain [,,,,], with lack of trust by clinicians and inexperienced end users in using AI within their routine workflows being 2 barriers reported in these studies.

Table 4. Consolidated view of research question 1B (barriers to artificial intelligence [AI] implementation)a.Horizon and categorySourceStudies, n (%)
Previous studiesThis study
Horizon 2: creating AI analytics
Process15 (58)

Barrier 1: insufficient performance assessmentFujimori et al []
Jauk et al []
Soltan et al []
Strohm et al []
Davis et al []
Allen et al []



Barrier 2: lack of standardized guidelines for AI implementationSharma et al []
Wolff et al []
Chomutare et al []
Pou-Prom et al []
Sendak et al []
Soltan et al []
Strohm et al []
Svedberg et al []



Barrier 3: disruptive integrationLee et al []
Chomutare et al []
Joshi et al []
Baxter et al []
Sandhu et al []
Sendak et al []



Barrier 4: inadequate continuous learningKanakaraj et al []
Sendak et al []
Wang et al []



Barrier 5: complexity of maintenanceWolff et al []
Chomutare et al []



Barrier 6: lack of clear consensus on alert definitions—b


Barrier 7: insufficient data preprocessing—

Information8 (31)

Barrier 8: poor data qualityLee et al []
Wolff et al []
Chomutare et al []
Joshi et al []
Jauk et al []
Pou-Prom et al []
Eche et al []
Gruendner et al []



Barrier 9: data heterogeneity—Pantanowitz et al []
Wilson et al []



Barrier 10: data privacy—


Barrier 11: challenges with data availabilityLee et al []
Wolff et al []
Chomutare et al []
—

Technology5 (19)

Barrier 12: lack of customization capability—Baxter et al []
Blezek et al []
Sohn et al []



Barrier 13: computational limitations of hardware—Fujimori et al []
Gruendner et al []

Horizon 3: implementation of new models of care
People5 (19)

Barrier 14: inexperienced end users with AI outputJoshi et al []
Sandhu et al []
Wilson et al []



Barrier 15: lack of clinician trustLee et al []
Chomutare et al []
Fujimori et al []
Sandhu et al []
Strohm et al []


Process2 (8)

Barrier 16: alert fatigueLee et al []
Chomutare et al []
Joshi et al []
Sandhu et al []



Barrier 17: difficulties with understanding AI outputs—

Information1 (4)

Barrier 18: data shift

aBarriers identified in previous reviews and this review were mapped to 4 categories of the 3-horizon framework [].