Study selection

Figure 1 illustrates the article selection process. The search strategy yielded 1836 articles after removing duplicates, from which we reviewed the full text of 133 articles against the defined inclusion criteria. We excluded 125 of these for the following reasons: was a duplicate (n = 1); not an original study (n = 1); primary care setting (n = 4); CDSS was developed but not implemented (n = 20); or CDSS implementation was not costed (n = 99). A total of 8 remaining articles were included in the review [23–30]. Forward citation searching (n = 166) and backward citation searching (n = 240) were conducted on the 8 included papers resulting in the identification of 1 eligible paper that subsequently met the inclusion criteria [31]. Therefore, 9 papers were included in this review.

Fig. 1

PRISMA flowchart of study selection process. Study selection was in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines for systemic reviews [14], and the flowchart is sourced from [22]

Study characteristics

Key characteristics of the included studies are summarised in Table 1. All except one of the nine included studies were from high-income countries. Two included papers conducted prospective studies across multiple centres [30, 31]. All other studies were retrospective observational single-centre designs that described implementation within a single service [23–29].

Table 1 Characteristics of included papers

All articles were conducted within the context of a health economic study design. Four of the studies were cost-effectiveness analyses [23, 27, 29, 31], three were cost analyses [25, 26, 28] and two were cost-benefit analyses [24, 30]. Economic evaluations took the perspective of the health service [23, 27–29, 31] or was not specified [24–26, 30]. None of the articles reported using an implementation science theory, model or framework to assist CDSS implementation.

CDSS characteristics

The type of CDSS implemented varied across the included studies. Three papers included an alert or early warning system which aided in the identification of patients at risk according to the respective clinical guideline [23, 24, 26]. For example, one of the papers investigated a paediatric early warning system that used a multicomponent scoring tool with an action algorithm to identify hospitalised children with clinical deterioration [24]. Three papers implemented a computerised provider order entry (CPOE) system which aims to reduce medication errors and adverse drug events by minimising illegible writing, unstructured orders and dosing variability [27, 30, 31]. Another study implemented order sets which are a bundle of clinical orders grouped together to improve adherence to clinical guidelines [23]. Three papers implemented a management system for specific clinical contexts. Management systems are multi-functional, typically multidisciplinary and often integrate CPOE systems. A patient data management system was investigated in an included paper which supported bedside clinical documentation. Its functionalities included data acquisition from monitoring and medical devices, bedside CPOE facilitating calculations of drug doses and fluid balances, import interfaces from laboratory/microbiology/radiology data and surgery reports, automated calculation of ventilation times, automated scoring and semiautomated coding of diagnoses/procedures with interface for exporting data directly to the electronic billing system [25]. Another included study implemented a patient blood management system which was defined as, "an evidence-based, multidisciplinary approach to optimising the care of patients who might need transfusion". It aimed to ensure optimal treatment was given while reducing avoidable/inappropriate use of blood and blood components [28]. Electronic medication management system (eMMS) was the final management system studied in an included paper. The eMMS interfaced with the existing CPOE allowing doctors to prescribe medication electronically as well as having functionalities to alert for drug allergy checking, pregnancy warnings, therapeutic duplication and some dose-range checking [29].

CDSS implementation strategies and cost

Implementation strategies were extracted from the information contained in the included papers and categorised against the ERIC framework. Identified implementation strategies were then cross-checked with reported costs in the included papers to determine if the strategy had been costed. A summary of these findings is presented in Table 2. Figure 2 provides a high-level illustration on the number of papers that costed, partially costed or did not report costs for each cluster of implementation strategies.

Table 2 Implementation strategy categorisation and determination of whether it was costed in the included paperFig. 2

Costing of the Expert Recommendations for Implementing Change (ERIC) clusters in the included papers. The figure illustrates the number of included papers that reported a discrete implementation strategy categorised to an ERIC implementation [19]. Within individual papers, each ERIC cluster was determined to be ‘costed’, ‘partially costed’ or ‘not costed’

Use of evaluative and iterative strategies

Eight papers reported ‘using evaluative and iterative strategies’ [23–29, 31]. The most common implementation strategy in this cluster was stage implementation scale-up which was conducted in five of the papers [24, 25, 27, 29, 31] and costed in four papers [25, 27, 29, 31]. Stage up ranged from 6 months [24] to 6 years [29]. Audit and provide feedback [23, 28] and purposely re-examine the implementation [24, 31] were implementation strategies present in more than one included paper and consistently costed. Agulnik et al. were the only paper to partially cost the strategies reported under this cluster: the purposeful re-examination of implementation was costed, but the paper did not report costs for the three other strategies (the assessment for readiness and identification of barriers and facilitators, staging implementation scale-up and development and organised quality monitoring system) [24].

Provide interactive assistance

Seven papers ‘provided interactive assistance’ [23–25, 27, 28, 30, 31]. Centralise technical assistance was the most common implementation strategy in this cluster and was costed in all papers [27, 28, 30], except one [25]. Centralised technical assistance included centralised IT staff to maintain the CDSS [25], help desk support [27, 30] and "a member of the [electronic patient record (EPR)] staff was responsible for implementing it on the EPR system" [28]. Three papers reported providing clinical supervision, and all papers provided costs for the strategy [23, 24, 28].

Train and educate stakeholders

Seven papers reported strategies in the ‘train and educate stakeholders’ cluster, and all seven papers conducted ongoing training as an implementation strategy [23, 24, 27–31]. Training included: nursing skills fair [23], 30-min training workshop [24, 28], and annual clinical personnel training [29]. Three papers did not provide details of the training [27, 30, 31]. Most papers costed ongoing training except for two papers that partially costed it by reporting the costs for some clinicians (including pharmacists, nurses) but not for physicians [24, 30]. The final paper that partially costed implementation strategies reported under this cluster was Swart et al.; ongoing training and educational meetings were costed, but train the trainer was not [28]. Develop educational materials was present in more than one included paper and was consistently costed [23, 24].

Develop stakeholder interrelationships

Six papers ‘developed stakeholder interrelationships’ by employing five different implementation strategies [23, 24, 27–30]. Identifying and preparing champions was a common strategy in this cluster and was consistently costed across the papers [24, 28, 30]. Champions and the cost allocations included the followng: a nurse educator with 50% salary support [24], a transfusion practitioner at 0.8 full-time equivalent (FTE) [28] and a physician champion with an unclear cost allocation [30]. Only two papers partially costed this cluster [23, 24]. Afshar et al. costed two strategies (use advisory boards and workgroups and involve executive boards) but did not cost the final strategy (use an implementation advisor) in this cluster [23]. As mentioned, Agulnik et al. costed identifying and preparing champions but did not cost two other strategies: model and simulate change and advisory boards and workgroups [24].

Adapt and tailor to context

Five papers