Engagement is a necessary condition to test audit and feedback design features: results of a pragmatic, factorial, cluster-randomized trial with an embedded process evaluation

Our methods are described in detail in the published protocol [23] and are summarized here in accordance with the CONSORT Checklist for cluster randomized trials (Additional file 1). There were some deviations from the protocol, which are outlined below.

Trial design

This was a 2 × 2 factorial, pragmatic, cluster-randomized trial with an embedded process evaluation. The trial is registered on ClinicalTrials.gov (NLM identifier: NCT02979964).

Setting

This trial took place in the province of Ontario, Canada. Ontario Health (OH—formerly Health Quality Ontario at the time of the study), the provincial advisor on quality in healthcare, supports quality improvement through various initiatives. One such initiative is their “Practice Reports,” whereby confidential, aggregate feedback is offered to physicians across the province, combined with change ideas for quality improvement. These reports are populated using data from existing administrative health databases and are developed using the best evidence, established methods, and stakeholder advice. The senior author (NMI) established the Ontario Health Implementation Laboratory (OHIL), a partnership with OH to support the optimization of their A&F initiatives [13, 14]. This trial focused on OH’s “MyPractice: Long-Term Care” report (http://www.hqontario.ca/Quality-Improvement/Guides-Tools-and-Practice-Reports/Long-Term-Care) which provides physicians with feedback about their prescribing of medications for nursing home residents which potentially increase their risk of falls. At the time of the study, physicians had to opt-in to receive a feedback report and log into a system in order to access it.

Participants

Eligible physicians were those working in the nursing home sector in Ontario who had (i) voluntarily signed up to receive their report prior to randomization and (ii) consistently had > 5 residents that they cared for in the nursing home setting (to allow for adequate data capture). The participating research ethics boards approved a waiver of consent with the provision of opt-out opportunities.

Interventions and mechanisms of action

Full details of the history of the report and its re-design in preparation for this trial were reported previously [23]. The re-design process involved remote usability testing employing think-aloud methods to inform report optimisation [23]. Two report features were manipulated in this trial: (i) the benchmark used for comparison and (ii) information framing.

Manipulated feature 1—The benchmark

Previously, the report compared physicians’ data to provincial and regional averages. OH felt that a benchmark of the top 10% of peers used in previous research [6,7,8] may risk unintended discontinuation of appropriate medications. The top quartile was considered acceptable for the purposes of the trial while avoiding unnecessary harms to residents. Participants’ prescribing rates were therefore compared to either the median prescribing rate among Ontario physicians (Ontario median) or among physicians with the lowest prescribing rates (Ontario top quartile).

Manipulated feature 2—Information framing

We developed a “risk-framed” and a “benefit-framed” version of the report. The risk-framed version focused on the proportion of residents prescribed high-risk medication. Risk-framing was presented visually (a graph demonstrating the percentage of patients at risk, with red colouring), and in text form (“n additional/fewer resident(s) in my practice may be/are at increased risk associated with (medication)”). The benefit-framed version indicated the proportion of residents for whom high-risk medications were avoided, using a graph demonstrating the percentage of patients safe from risk, with green colour emphasis, and using the statement “n additional/fewer resident(s) in my practice may be/are safe from risks associated with (medication)”. Both versions were refined iteratively through user-testing [23].

Thus, four variants of the report were developed (excerpts included in Additional file 2): (i) Ontario median comparator with benefit-framing, (ii) Ontario top quartile comparator with benefit-framing, (iii) Ontario median comparator with risk-framing, and (iv) Ontario top quartile comparator with risk framing. Full details of our overall program theory are included in the protocol [23]. We hypothesized that greater improvements in practice would be achieved when feedback recipients were compared to the top quartile and when information was framed to emphasize risks of harm. Proposed theory-informed mechanisms of action are outlined in Fig. 1 [5, 24,25,26,27].

Fig. 1figure 1

Proposed theory-informed mechanisms of action of the two factors varied in the Audit & Feedback report

Outcomes

The primary outcome was the mean number of central nervous system (CNS)-active medications per resident per month, with the primary endpoint for analysis being 6 months post-intervention. CNS-active medications included antipsychotics, opioids, benzodiazepines, and antidepressants (including tricyclic antidepressants and trazodone), consistent with the indicator used in the OH report. We selected this as the primary outcome to enable us to capture any prescribing changes directly influenced by the report indicators. We planned to assess antipsychotic and benzodiazepine prescriptions as secondary outcomes, as well as statin prescriptions (as a non-targeted control or “tracer outcome” [28]) [23]. However, we did not conduct these analyses due to poor report engagement (as described in the results).

Data collection

In this pragmatic trial, we used provincial health administrative data to assess baseline characteristics and outcomes. Data were compiled from (1) the Ontario Drug Benefits database, which covers nearly all prescriptions in nursing homes; (2) the Canadian Institute for Health Information databases covering all inpatient hospitalizations and emergency department visits; (3) the Ontario Health Insurance Plan database, covering physician billings; (4) the Registered Persons Database covering demographic information; and (5) the Continuing Care Reporting System database for clinical and demographic information on nursing home residents collected using the Resident Assessment Instrument (RAI). A full RAI assessment completed by nursing home staff is legislatively mandated within 14 days of admission and updated annually or with a change in status; a quarterly RAI assessment is required every 92 days. RAI data were used to identify dates of admission and discharge to define the appropriate set of residents contributing to each time period. For each 3-month period under investigation, residents were assigned to a most responsible physician according to previously defined algorithms [29].

We used the RAI for demographic and clinical characteristics of residents, including clinical assessment scores (e.g., function scale, pain scale, depression rating score, aggressive behaviour score). We used Ontario Health Insurance Plan data to determine whether residents had a specialist consultation in the prior year by a geriatrician or psychiatrist. We used the Canadian Institute for Health Information datasets to assess whether residents had an emergency department visit in the prior year (using the National Ambulatory Care Reporting System database) and whether residents had a hospital admission in the prior year (using the Discharge Abstract Database). These databases provide complete population-level data for the variables of interest.

Randomization

To prevent contamination due to physicians working across multiple homes, the unit of randomization was groups of one or more nursing homes sharing physicians. All eligible physicians were included in the clusters. An independent statistician randomized these clusters independently to the two factors (resulting in four experimental conditions), stratifying by a total number of nursing home beds in the cluster [30], using a randomly permuted block design of length four. The randomization list was provided to OH for the purposes of distributing the reports and was not accessible to any others on the research team besides the statistician.

Sample size

We anticipated having approximately 160 clusters, with an average of 350 beds per cluster. In a 2 × 2 factorial design assuming no interaction and similar effects for each factor, a test of each intervention at 6 months in an ANCOVA design would achieve 90% power to detect an absolute mean difference of 0.3 in the primary outcome (i.e., a difference in the mean number of CNS-active medications per month of 3 versus 2.7). Based on previous data, we assumed a standard deviation of 4, an intracluster correlation coefficient of 0.05, a cluster autocorrelation of 0.8, and an individual autocorrelation of 0.9 [31].

Blinding

Participants were not explicitly blinded, but the risks of this were felt to be minimal, given that the physician were not aware of the variations being tested nor the outcome measures. The analysts were blind to allocation status.

Data analysis

Descriptive characteristics of nursing home residents included variables assessed as part of the RAI assessment: therefore, only those residents for whom a recent RAI assessment had been completed were included in the analysis of resident characteristics at baseline. Primary outcome analyses included the broader population of included physicians’ residents. All primary analyses were by intention-to-treat and compared the four arms. The primary outcome was analyzed using a general linear mixed effects regression model; time was specified as a continuous variable, and a common secular trend was imposed across all study arms with the effect of the intervention modelled as a slope deviation from the trend. The analysis adjusted for the size of each home (number of beds) as a fixed effect. A random intercept and slope for time were specified for the unit of randomization (group of homes). The primary comparison between the arms at 6 months post-intervention was estimated using least square mean differences, together with 95% confidence intervals. Factorial analyses were conducted as a secondary analysis because the traditional approach (i.e., an interaction test followed by dropping the interaction term if non-significant) can lead to bias in factorial trials [32]. OH provided data on report engagement (number of physicians who downloaded the report). Due to the poor engagement with the report (as described in the results), we did not conduct the additional analyses outlined in the protocol, including the economic evaluation.

Process evaluation

Physicians who downloaded their report were sent an email invitation to complete a questionnaire which assessed the proposed mechanisms of action outlined in Fig. 1. The questionnaire included one item measuring each of these mechanisms in relation to prescribing three classes of high-risk medications and the tracer outcome (benzodiazepines, antidepressants, antipsychotics, and statins). Each item was scored using a five-point Likert scale. We compared construct scores for each of the trial factors using independent samples t tests. We planned to use mediation analyses to determine whether interventions worked through hypothesized pathways [23]: however, our small sample size precluded this investigation. All analyses were conducted using SPSS.

Questionnaire participants who indicated interest were invited to take part in a telephone interview. The interview topic guide focused on report use and ideas for improvement; prioritization of behavior change in relation to the prescribing indicators in the report; and the hypothesized mechanisms of action. Interviews were audio-recorded, then transcribed verbatim by an external third party. Analysis was conducted in NVIVO 10 and informed by the framework analysis method [33, 34]. An initial coding framework was developed, and constructs from the Consolidated Framework for Implementation Research [35] were added as this incorporated themes which developed from open coding. Refinement of themes involved study team discussions as necessary.

留言 (0)

沒有登入
gif