2.1 Study Design, Setting and Population

A multi-site before and after retrospective audit was undertaken at three public hospitals in New South Wales, Australia. Hospitals A and B were regional acute hospitals with 250 and 300 beds, respectively. Hospital C was a metropolitan principal referral hospital with 820 beds. Data were collected from study hospitals before and after the implementation of each hospital’s eMM system. The eMM allowed for electronic prescribing, review and administration of medications, but included no specific CDS for DDIs. Thus, the prescribing of two potentially interacting medications did not result in the triggering of a DDI alert for prescribers. The same eMM system was introduced across hospitals A and B (Cerner Millennium, https://www.cerner.com/solutions/health-systems), whilst hospital C implemented a different system (MedChart, https://www.medchart.com/).

Study patients were randomly selected from all admitted patients during two time periods: pre- and post-eMM implementation (Fig. 1). Patients were excluded if (i) their hospital stay was longer than 3 months, (ii) they were in a ward/unit without eMM being implemented or with a different eMM system (e.g. chemotherapy units, intensive care units and some psychiatric wards) or (iii) they were in ambulatory care to receive regular intravenous infusions, for example, blood or blood product transfusions or dialysis. In addition, anaesthetic charts were excluded from chart review due to variations in documentation of anaesthetic medications across the study sites. Ethics approval was obtained from one of the hospitals’ Human Research Ethics Committee (reference no.: 8/02/21/4.07), and site-specific governance approval was subsequently obtained from all sites.

Fig. 1

Study design and hospitals (eMM, electronic medication management)

2.2 Drug–Drug Interaction (DDI) Classification—Potential and Clinically Relevant DDIs

All medication orders that were active on a patient’s medication chart on the same calendar date were screened for DDIs using Stockley’s Drug Interactions Checker [17], a standard international reference used in Australian hospital practice. All moderate or severe DDIs specified by Stockley’s were classified as a potential DDI (pDDI). This term was used to distinguish DDIs from cDDIs.

The clinical relevance of each pDDI, that is, the probability and potential for harm to each patient, was assessed by trained clinical pharmacists. The pharmacists classified the likelihood of clinical relevance as either very unlikely, unlikely, possible or likely. In this review process, they identified 26 contextual factors, including 11 drug factors, 11 patient factors, two settings factors and two other factors that determined whether a pDDI was clinically relevant. See Online Resource Table S1 for detailed definitions and examples of these contextual factors that were considered when determining clinical relevance of each pDDI. All pDDIs with possible or likely clinical relevance were categorised as cDDIs. A step-by-step workflow of the DDI assessment process is presented in Online Resource Fig. S1.

2.3 Data Collection

Clinical pharmacists who were trained in the procedures described in the study protocol reviewed medication orders and medical records [16]. Five clinical pharmacist reviewers undertook chart review across the three study hospitals. All five reviewers were experienced hospital pharmacists, but were independent from study hospitals. Reviewers documented patient demographics including date of birth, sex, admission and discharge date and diagnoses. For each patient reviewed during the pre-eMM period, details of every medication prescribed during the patient’s admission were recorded, including drug name, route, dose, frequency and start and end dates. For post-eMM patients, medication data were extracted from eMM systems.

Reviewers collected information on whether each drug in a DDI pair was administered. A drug was recorded as ‘not administered’ if the drug was not administered at all during the admission.

2.4 Inter-Rater Reliability Assessment

Inter-rater reliability testing between the five pharmacist reviewers was conducted before commencing independent data collection. This was to ensure that all reviewers were consistent in identifying contextual factors related to clinical significance and classifying the likelihood of clinical relevance of pDDIs. The reviewers were required to achieve a minimum Cohen’s Kappa score of 0.7 on contextual factor categorisation and 0.8 on clinical significance level, indicating moderate-to-strong level of agreement [18], before starting independent data collection.

2.5 Harm Assessment

Medication charts were reviewed by the study pharmacists to determine whether medications involved in a cDDI were administered to patients concomitantly. The clinically relevant time frame of two interacting drugs was assessed by pharmacist reviewers with respect to the pharmacokinetic and pharmacodynamic characteristics of each agent on the basis of published online databases (MIMS Australia [19] and Micromedex [20]). If both medications were administered within an estimated clinically relevant time frame, pharmacist reviewers proceeded to identify whether there was evidence that the possible harms associated with the cDDI, as described in Stockley’s, had manifested in the patient. An in-depth review of patient records was undertaken to identify signs of possible harm to the patient from the cDDI during that hospital admission. This included, for example, abnormal findings from clinical examinations or vital signs, pathology results, patient symptoms or interruption to a treatment plan (e.g. prescribing of an antidote). For those patients who were identified to have experienced potential harm, study pharmacist reviewers prepared case studies detailing all relevant information. These case studies were reviewed by two clinical pharmacologists (physicians) to determine whether actual harm had occurred from cDDIs. On the basis of published classifications of DDI severity and resulting potential harms [21, 22], the clinical pharmacologists independently reviewed the material and made two assessments: (i) plausibility that the cDDI caused the identified harm (unlikely, possible, probable or certain) and (ii) severity of harm experienced (no harm, minor, moderate, serious or severe). Definitions of each category are in Online Resource Tables S2 and S3. The independent assessments were compared. Panels, involving the clinical pharmacologists and study pharmacist reviewers, were convened to discuss all disagreements and reach consensus on plausibility and severity of patient harms in all cases.

2.6 Statistical Analysis

Descriptive summaries of patient demographic and clinical characteristics were produced for both pre- and post-eMM periods. Summary statistics on pDDIs and cDDIs were presented, including contextual factors identified when classifying pDDIs as either cDDIs or non-cDDIs. The distributions of pDDIs and cDDIs per admission by age, sex and eMM were displayed in clustered box plots. The same plots were created for the distributions of pDDIs and cDDIs per ten unique drugs prescribed during hospital stay, that is, per ten active ingredients per admission. Among the pDDIs identified from patient records, the primary outcome examined was occurrence of a cDDI. A cDDI was classified as leading to actual harm when its plausibility was rated as probable or certain, and severity as minor or above. Sensitivity analyses were conducted and presented using different levels of harm plausibility and severity.

To examine the effect of the introduction of eMMs without DDI decision support, three multilevel regression models were applied to three outcomes of interest (i.e. dependent variable of each model): (1) the occurrence of a cDDI, (2) the administration of both medicines identified as contributing to cDDIs and (3) the occurrence of actual harm. Mixed-effect multivariable logistic regression models were developed with consideration of the correlation of multiple DDIs for the same patients. Study period (pre- versus post-eMM) was an explanatory variable of interest. Other explanatory variables included study hospital, patient’s age group, number of unique drugs prescribed during the admission and number of medication orders and relevant contextual factors identified by the pharmacist reviewers. These variables in the multivariable models were selected on the basis of prior knowledge to control for potential confounding. Backwards elimination and forwards addition steps were used to arrive at final models. Total number of medication orders was excluded from final models due to collinearity with number of unique drugs prescribed. Number of unique drugs prescribed was found to have a linear association with the outcome and thus kept as a continuous variable. Correlation and variance inflation factors were examined, and no other collinearity was detected in final models. All statistical analyses were performed using SAS 9.4.