Study overview

This pragmatic implementation trial is designed to test the comparative effectiveness of different types of alerts and notifications within the EHR. More specifically, we propose two different interventions for improving DOAC prescribing. The first intervention, intended to target new inappropriate DOAC prescriptions, is an automated EHR alert that occurs at the time a DOAC medication is prescribed but some potential error exists (e.g., drug-drug interaction, wrong dose for given renal function). All eligible prescribers will be randomized with equal probability to receive either a detailed alert, or the same detailed alert that also includes a referral link for optional DOAC pharmacist review.

The second intervention, intended to target existing inappropriate DOAC prescriptions, is an EHR notification. Prescribers with ownership of an inappropriate existing DOAC prescription (i.e., a prescription identified by our system as inappropriate any time after the DOAC medication is prescribed when a new potential issue develops [e.g., worsened renal function that impacts dosing, new drug-drug interactions]) will be randomized with equal probability for the notification to be routed either to the prescriber’s inbox or directly to the anticoagulation pharmacist for follow-up.

Aims and objectives

Primary aim: Our study’s primary aim is to determine the effect of the notifications targeting existing inappropriate DOAC prescriptions on the proportion of inappropriate DOAC prescriptions that are changed within 7 days. Our primary aim hypothesis is that notifications that are routed directly to the pharmacist rather than the original prescriber will result in a higher proportion of inappropriate prescriptions changed within 7 days.

Secondary and exploratory aims:

• Aim 2 will examine the effect of the alerts targeting newly prescribed inappropriate DOACs. In Aim 2a, we will examine the overall proportion of alerts that result in a prescription change within 7 days, without accounting for the type of alert received. In Aim 2b, we will examine whether the type of alert (i.e., alert only or alert + pharmacist referral) resulted in a higher proportion of inappropriate DOAC prescriptions changed within 7 days.

• Aim 3 will examine changes in the magnitude of effects over time for both alerts and notifications. In aim 3a, we will examine the change in the proportion of existing inappropriate DOAC prescriptions that trigger notifications that are changed within 7 days over the 18-month course of the study, both overall and by condition. In aim 3b, we will do the same for new DOAC prescriptions that trigger alerts.

• Exploratory analyses will examine potential moderators for alerts and notification conditions, to understand whether there are certain prescribers or patients that benefit most from alerts or notifications that encourage pharmacist collaboration. We will also examine implementation outcomes including patient reach for all types of alerts and notifications, variation in prescriber engagement with alerts/notifications that encourage pharmacist collaboration, and fidelity of prescription changes. Finally, we will assess the effect of the entire system of notifications and alerts on the prevalence of patients with the health system that are receiving inappropriate DOACs.

Setting

This pragmatic prospective randomized trial will be conducted within one health care system, Michigan Medicine. Michigan Medicine includes more than 4000 clinicians who provided care in over 2.6 million patient clinic visits in fiscal year 2022. The anticoagulation clinic includes four pharmacists and ten nursing staff members who provide care to more than 3000 warfarin-treated patients. In 2020, the anticoagulation clinic staff began monitoring for appropriate DOAC prescribing using a dashboard built within the EHR. A single-day (December 21, 2020) cross-sectional analysis at Michigan Medicine found 9325 patients had DOAC use documented by 1002 primary care, cardiology, hematology, or surgery prescribers (median 23 patients/prescriber, interquartile range [IQR] 7–47). Of these, 670 (7.2%) patients (among 250 unique prescribers) did not follow evidence-based guidelines, with a median of 2 (IQR 1–5) unsafe DOAC prescriptions per prescriber observed on that single-day snapshot.

Implementation strategies

We will use the following implementation strategies to encourage evidence-based DOAC prescribing. Each strategy is described below, from the perspective of the prescriber, using the Proctor et al. [21] framework (Table 2).

Table 2 Implementation strategiesEHR medication alerts/notifications

• For new prescribing errors, prescribers will be shown an EHR alert immediately upon entry of a new prescription that does not meet current evidence-based guidelines. Alerts were designed through a user-centered design process [17] to ensure they are clear and usable. All alerts inform the prescriber of the potential reason for inappropriate prescribing (e.g., drug-drug interaction) and recommended actions the prescriber can take (e.g., ordering an alternative DOAC or another drug) (Fig. 2).

• For existing prescription errors, notification messages alerting personnel to an inappropriate DOAC prescription, as well as clear recommendations for what changes to make to the prescription to align with evidence-based guidelines, will be sent once a day. These notifications were also designed using a user-centered design process (Fig. 2).

Fig. 2

Example dialog box from the alert. Example shows drug-drug interaction between the DOAC rivaroxaban and dronedarone

Prescriber-pharmacist collaboration

Our second strategy for increasing the implementation of evidence-based DOAC prescribing is encouraging prescriber-pharmacist collaboration. This is done in conjunction with the EHR alerts and notifications, described above, but entails different actions. For new prescription alerts, we encourage prescriber-pharmacist by offering a button for prescribers to click within the medication alert that will trigger a request for a DOAC pharmacist to review the prescription (and patient case) and recommend any necessary changes. For existing prescription notifications, prescriber-pharmacist collaboration is encouraged by routing notifications of inappropriate prescriptions initially to DOAC pharmacists directly, rather than prescribers. The pharmacist will then use their expert judgment to determine when it is clinically appropriate to contact the prescriber and request or recommend a prescription change.

Trial study design

This study is an 18-month prospective randomized clinical trial for alerts and for notifications. Prescribers will be randomized to different types of alerts and/or notifications when a new or existing DOAC prescription is determined to not meet current evidence-based guidelines.

For new inappropriate DOAC prescriptions during the study period: providers who write a new DOAC prescription that does not meet current evidence-based guidelines will be randomized at first instance with equal probability to one of two types of new prescription EHR alerts:

• Alert style 1 will include information about why the prescription is inappropriate as well as recommendations for changing to an evidenced-based prescription.

• Alert style 2 will also include this same information, but will also include a “button” that can be clicked to refer the prescription to a DOAC pharmacist for review.

For existing DOAC prescriptions that are identified as inappropriate, prescribers will be randomized at first instance with equal probability to one of two routings for notifications:

• Notification routing A will route to the prescriber to review the prescription and change it as appropriate.

• Notification routing B will route directly to the DOAC pharmacist for review. The pharmacist may then opt to change the prescription themselves or consult with the prescriber about possible changes.

Note that for our implementation strategies specified above, all four alerts and notifications make use of well-designed EHR alerts; alert style 2 and notification routing B augment this strategy with the encouragement of pharmacist-prescriber collaboration.

Eligible prescribers may be randomized once per condition (alerts, notifications) during the 18-month study duration for alerts and for notifications, immediately following EHR identification of their first inappropriate new (for alerts) or existing (for notifications) DOAC prescription. Prescribers will continue to receive their assigned alert and/or notification type for subsequent inappropriate prescriptions for the duration of the trial. As such, while the trial for alerts and for notifications will be active for the trial in the Michigan Medicine EHR for a total of 18 months, prescribers will not be randomized or receive alerts/notifications until their first inappropriate prescription is identified. A waiver of documented informed consent was approved by the Michigan Medicine Institutional Review Board (IRB). Prescribers were notified prior to the alerts and notifications going live and can opt out at any time during the study period.

Eligibility and recruitmentPrescribers

All Michigan Medicine clinicians with prescribing privileges (including attending physicians, house officers, nurse practitioners, and physician assistants) who see patients in the ambulatory setting and who are not members of the study team will be eligible for study enrollment. Prescribers will be enrolled in the trial and randomized upon having an ambulatory patient whose DOAC prescription triggers an alert (either initial or longitudinal). Prescribers who do not have a new or existing DOAC prescription that is identified as inappropriate during the 18-month trial duration will not be considered eligible for the trial and thus not randomized. We anticipate that 300 prescribers will be enrolled in the trial. All prescribers were notified of the new alert system and companion trial through official EHR communication channels before initiating the trial. Prescribers may opt out of participation at any time.

Prescriptions/patients

Our system for assessing DOAC prescription appropriateness will include DOAC prescriptions written in an ambulatory setting for all adult (age ≥ 18) patients who see an eligible prescriber. This includes patients that were initially prescribed a DOAC before the trial commencement but develop unsafe use during the study period. Prescriptions will be excluded if they were written in the Emergency Department or hospital setting (including upon discharge) or in a skilled nursing facility or another institutionalized setting, as prescribers in these settings typically do not follow patients longitudinally and therefore would not be appropriate targets for notifications when prescribing issues develop after the initial prescription is written. Further, inpatient DOAC prescriptions already undergo pharmacy review.

Comparison to current standard of care

At present, the Michigan Medicine system provides alerts for inappropriate DOAC prescriptions only for new prescriptions and only for select drug-drug interactions (no alerts specific to indication or renal/liver dysfunction). Furthermore, the currently existing alerts do not guide corrective actions. At minimum, our new system will improve upon this by (1) offering enhanced alerts that provide prescribers with additional information on the source of the inappropriate prescription and have been informed by a user-centered design process, and (2) adding existing prescription notifications that ensure that one or more medical professionals are notified when changes in patient characteristics affect the current DOAC prescription appropriateness.

To further ensure patient safety, study investigators have also created a safety review mechanism that will review any un-addressed unsafe DOAC prescribing to ensure that patients are not unnecessarily harmed. This will include a review of any non-evidence-based DOAC prescription at 14 business days following either an alert or notification.

Randomization

Randomization will occur at the prescriber level and separate, independent randomizations will be performed for new prescription alerts and existing prescription notifications at the time of EHR flagging prescribers' first instance of each. Prescribers may be randomized for alerts, notifications, both, or neither. Once randomized (i.e., after the first alert or notification), alert/notification types for that prescriber will remain consistent throughout the remainder of the study period.

To account for variation in both overall and inappropriate DOAC prescribing occurrence, randomization will be stratified by trainee (vs. non-trainee) status and specialist (e.g., cardiology, hematology) vs. primary care for non-trainees. A permuted block randomization implemented via computer scripts that can interface directly with the EHR will be used to generate stratified random assignments.

This trial is not fully blinded. Prescribers will be aware of the specific content of their assigned alerts and/or notifications but will be unaware of other intervention options. All study staff monitoring outcomes data collection will be blinded to treatment assignment.

Fidelity monitoring of alerts and notifications

To ensure fidelity of our alert/notification system, a random sample of alerts and notifications will be manually audited quarterly to ensure that they occur when clinically appropriate and contain correct clinical information. Any updates to evidence-based DOAC prescribing guidelines that would affect EHR alert logics (e.g., new drug-drug interactions) will also be continuously monitored, with alert logic updated for all prescribers at least monthly.

Data safety and monitoring board

The study Data Safety and Monitoring Board (DSMB) will monitor for appropriate clinical management decisions made by prescribers and pharmacists every 6 months during the study period. The DSMB will not report the results of the individual analyses to the study team, but rather will make one of the following recommendations based on their analysis of the data: (1) continue the study without any intervention, (2) provide re-education efforts to both prescribers and pharmacists, or (3) terminate the study due to a concern for patient harm.

Data

All data used to evaluate alerts and notifications will be pulled from the Michigan Medicine EHR through custom-built reports supplemented by automated and manual chart review. Data will be collected over the 18-month study timeline for alerts and for notifications. Only the trial PIs (Smith, Barnes) and data analysts will have access to the final trial dataset.

Outcomes and measures

EHR data collection will collect metrics for measuring key outcomes that align with the reach, effectiveness, adoption, implementation, and maintenance (RE-AIM) implementation framework [22], including the adoption of DOAC prescription changes by prescribers following receipt of an alert or notification (primary outcome), reach, clinical effectiveness, prescription change fidelity, and maintenance, as well as some key exploratory outcomes (e.g., pharmacist workload).

Primary outcome (DOAC prescription changes)

Our primary outcome measure is the number (and proportion) of existing medication notifications that result in any prescription change within 7 days. Given that clinical complexity prevents every patient from having a clearly defined “correct” DOAC prescription, our primary outcome will assess for any change that is made to the prescription (e.g., dosage, frequency, medication) following delivery of the notification. The 7-day interval was selected to allow time for appropriate anticoagulation clinic pharmacist referral, review, and recommendation to occur.

Secondary outcomesPrescription changes for new medication alerts

The number (and proportion) of new medication alerts that results in any prescription change within 7 days. As with the primary outcome, we will assess for any change that is made to the prescription following delivery of the alert.

Clinical effectiveness of our alerts and notifications will be measured by 30-day rates of clinical adverse events, including major [23] and clinically-relevant non-major bleeding (CRNMB) [24] events, as defined by the International Society on Thrombosis and Haemostasis; new or recurrent VTE events; and stroke or systemic arterial embolic events. Adverse events will be captured using two Michigan Medicine-developed health informatics tools, DataDirect and the Electronic Medical Records Search Engine (EMERSE) [25], and independently adjudicated by two expert clinicians (with a third expert available when different opinions arise). DataDirect and EMERSE capture clinical data (e.g., notes, labs, imaging, procedure reports) and allow for rapid identification of populations based on granular clinical details (e.g., demographics, diagnosis, medication use) and/or via text-based searches of the medical record for key terms (e.g., “bleeding”, “stroke”) across pre-defined patient populations. The adjudicators will be blinded as to provider randomization. The 30-day time period was selected to minimize potential contamination from the safety review mechanism described above, and we anticipate that rates for adverse events will be low.

Exploratory outcomes

Reach will be measured by assessing the proportion of DOAC patients that trigger alerts/notifications for inappropriate prescriptions and the proportion whose alerts/notifications are corrected.

Implementation/fidelity of alerts/notifications will be defined as (1) for prescribers, how often they order the medication recommended by the alerts/notifications or not (and, for the latter, whether they provide a reason); (2) for pharmacists, how often they respond to referrals from alerts/notifications; and (3) for both prescribers and pharmacists, time from referral/notification to change or recommendation in EHR. Fidelity metrics will be collected via automated chart abstraction using EMERSE [25] and validated by manual chart review of a random selection of 20% of the alerts/notifications that result in prescription changes.

To measure maintenance and sustainment, we will assess over-time changes in reach and adoption outcomes over the full 18-month duration of the study for alerts and for notifications.

Pharmacist workload

To estimate the pharmacist effort required to manage alerts and notifications, we will calculate the total number of referrals to the anticoagulation clinic during the study period. This will be estimated as the number of referrals/alerts per 1000 DOAC prescriptions. We will then also use tools built within the EHR to measure the time (in minutes) from which a pharmacist views a new alert/notification until one of three actions occur: (1) change in medication prescription is made, (2) alert is dismissed (and reason documented), or (3) message is sent to prescribing clinician. Together, this will allow us to estimate the total pharmacist effort required to manage DOAC alerts/notifications through this system (calculated as full-time equivalents [FTE]) required for a given population size of DOAC-treated patients. To further validate these estimates, DOAC pharmacists will also be asked to document their time spent interacting with and acting upon alerts and notifications during two separate two-week periods using a time-tracking worksheet developed for this study.

AnalysesPrimary aim

The primary aim analyses will compare the main effects of the two types of longitudinal notification routings (A vs. B) on our primary outcome, the proportion of patients who have their DOAC prescription changed within 7 days. Mixed-effects logistic regression models will be used to model the probability of changed prescription, with fixed effects for notification type (A vs. B) and stratification variables (resident vs. non, primary care vs. specialist), and any patient-level characteristics that are unbalanced. A prescriber-level random effect will account for patient clustering within prescribers, and an unstructured covariance matrix will be used for residual errors. While we anticipate that few patients will appear in the data multiple times over the 18-month trial period, a patient-level random effect will also be considered, as appropriate.

Secondary aims

Aim 2 will examine the effects of the initial alerts for new inappropriate DOAC prescriptions. In Aim 2a, we will examine the proportion of new inappropriate DOAC prescriptions that were changed within 7 days, without distinguishing between alert types. Then, in aim 2b, using a similar modeling approach to that used in the primary aim, we will assess whether patients whose prescribers were randomized to alerts that included DOAC pharmacist referrals were more likely to have their prescription changed than those that received the simple alert (alert 1 vs. 2).

Aim 3 will assess the maintenance of our treatment effects by examining longitudinal change in the effect size for both notifications and alerts. For these analyses, we will extend our initial two-level multi-level model (patients nested in prescribers) used in aims 1 and 2 to a three-level multi-level model that also accounts for time since alerts/notifications were activated. These models will thus include all parameters in aim 1/2, a fixed effect for time in months (0 to 18) since the alerts or notifications were turned on, and an interaction between time and treatment.

Exploratory analysesModerators

As DOAC pharmacist time is a limited resource in health care systems, moderator analyses will be used to explore the prescriber- and/or patient-characteristics that most benefit from notifications and alerts that facilitate prescriber-pharmacist collaboration. We hypothesize that both alerts and notifications that engage pharmacists will be more effective at improving change in DOACs prescriptions when:

• Prescribers (1) are based in primary care vs. medical specialists and/or (2) have prescribed fewer DOACs in the six months preceding randomization; and

• Patients (1) are aged 70+; (2) have a VTE (as opposed to AF) diagnosis, as the former has more complex dosing with which many clinicians have less familiarity; (3) have 5+ concurrent medication prescriptions; and/or (4) have moderate or worse renal function (creatinine clearance ≤ 60 ml/min).

Moderators will be assessed by adding interaction effects between indicators for alerts/notifications and the moderator(s) of interest to the analysis models described in aims 1 and 2 above. All moderators will be examined individually initially and all tested moderators will be reported. Those that show both clinical and statistical significance will be considered for use in tailoring alert delivery. Exploratory analyses will be reported using 95% confidence intervals.

Implementation outcomes

Descriptive statistics will be analyzed for all pre-defined implementation outcomes, including reach, clinical effectiveness, system-level adoption, implementation, and maintenance. Exploratory bivariate analyses will also look for key sources of variation as they relate to the prescriber or patient characteristics, especially those identified as potential moderators. We will also repeat aims 1 and 2 analyses with clinical effectiveness (including adverse events) and implementation fidelity outcomes to assess whether there are any statistically or clinically significant differences across alert or notification types for any outcomes.

Additional details about additional exploratory analyses, the handling of missing data, sensitivity analyses are included in the Supplemental Appendix.

Power and sample size

This study is powered for our primary comparison, the comparative effectiveness of two types of longitudinal notification routings (A vs. B) on changes in patient prescriptions. We anticipate 300 prescribers will be randomized (150 per notification), and that prescribers will average two patients triggering a notification. Thus, assuming a patient n = 600, prescriber-level intraclass correlation (ICC) of 0.1, and α  = 0.05, we will have 94% power to detect a difference in the proportion of prescriptions changed of 0.40 vs. 0.55 (risk ratio of 1.45) for notification A vs. B. Power for aim 2 analyses of initial alerts will be the same. Power calculations for other outcomes are included in the Supplemental Appendix.

Trial status

This study was approved by the University of Michigan IRB on March 16, 2022. Initial alerts (and concomitant randomization procedures) for AF went live on August 1, 2022 in a pilot period. Initial alerts and randomization for VTE and longitudinal notifications for both AF and VTE are planned to go live in March 2023, at which point our 18-month trial period will begin for all alerts and notifications.