Despite pre-specification of outcome assessment times in pragmatic randomized trials, participant deviation from these times is commonplace. A key analytic concern is whether there are factors linked to the outcomes of interest that are driving these deviations. Off-schedule or completely missed assessments may be related to participants’ health status—such as poor health or feeling well enough to engage in other activities (e.g., work, school, caregiving). Differences between the health status at the observed assessment times and the health status at the pre-specified assessment times can result in overestimation or underestimation of the harms or benefits of study interventions [1].

The Patient-Centered Outcomes Research Institute® (PCORI®), a nonprofit organization established by the Affordable Care Act in 2010 in the United States, funded our group to develop and disseminate a statistical methodology to account for irregular (i.e., off-schedule or missed) assessment times in pragmatic randomized clinical trials. PCORI® values pragmatic randomized trials because they produce practical, actional evidence that can directly impact patient care and improve health outcomes in real-word settings. Irregular assessment times in these trials present a significant challenge for trialists in estimating unbiased treatment effects.

Our project focused on trials evaluating interventions to improve care for patients with chronic diseases and limited socioeconomic resources. Members of the investigator group (authors AA and JK) conducted four pragmatic randomized clinical trials: two enrolled adults with asthma (ARC [2]—NCT02086565 and HAP2 [3]—NCT 01972308), one enrolled children presenting with uncontrolled asthma to emergency departments (CHICAGO Plan [4]—NCT02319967), and the fourth enrolled adults hospitalized for heart failure, myocardial infarction, pneumonia, COPD, or sickle cell disease (PArTNER [5]—NCT02114515). All four pragmatic trials were designed to test interventions under real-world conditions and experienced substantial missingness and irregularity of assessment times (see Table 1 and Fig. 1); in ARC and HAP2, primary outcomes were assessed in clinic or by phone, depending on the preference of the participant, and in CHICAGO PLAN and PArTNER, primary outcomes were assessed by phone.

Distribution of actual vs. targeted assessment times for four studies. Treatment-specific distributions of the actual time of first (red), second (blue), third (green), and fourth (purple) scheduled assessments. Shaded regions reflect assessment windows

To provide first-hand insights into the problem, our project is informed by a diverse Stakeholder Advisory Board (SAB), consisting of two individuals from affected populations (i.e., people with asthma or other conditions relevant to the study protocol), clinicians, clinical trialists, implementation scientists, a qualitative researcher, and biostatisticians. Importantly, the researchers on the SAB have diverse expertise extending beyond the type of behavioral trials highlighted above, including substantial experience with trials outside the United States. The initial SAB activities were devoted to understanding, from the stakeholders’ perspective, what might cause patients to postpone or miss a pre-specified assessment. In a meeting of the entire SAB, members generated a list of reasons for off-schedule or missed assessments. The list reflected the diverse perspectives and experiences that each member brought to the elicitation process. Authors FKB and JDS created a coding tree to sort the reasons into discrete categories. The coding tree included codes for the five key domains (outer setting, inner setting, innovativeness of the trial, individuals involved, and implementation process) in the Consolidated Framework for Implementation Research 2.0 (CFIR2.0) [6] and subcodes describing who or what causes off-schedule or missed assessments in each of those domains. CFIR2.0 was selected because it provides a systematic approach for understanding the multiple domains (determinants) of study implementation, including various features of the study (e.g., pre-specified data collection schedule).

Figure 2 (top) displays the relationship among the domains, and Fig. 2 (bottom) provides examples of key constructs within each of the CFIR2.0 domains. Our insight from this process is that the timely completion of outcome assessments is a function of multiple determinants, only some of which are related to participants’ health status. Importantly, we identified determinants of irregular assessment times that can be modified during the protocol design stage (e.g., developing study protocols that are feasible for both participants and study implementers). At the protocol implementation stage, investigators can then reassess barriers and enact flexible mitigation plans to reduce the risk of irregular assessments (e.g., collection of outcomes via home visits, rather than depending on participant travel to clinics or study sites).

(Top) CFIR 2.0 domains to consider to optimize protocol adherence in a clinical trial. (Bottom) Examples of constructs within each CFIR 2.0 domain that affect protocol adherence in a clinical trials; stars indicate modifiable factors

Our preliminary observations stem from four localized behavioral trials conducted by researchers at two universities as well as the extensive experience of our SAB. We conjecture that our observations are generalizable to other pragmatic trials, including those that are geographically diverse, non-US-based and involve pharmaceutical treatments. Research to more formally evaluate this conjecture is warranted as well as studies to evaluate multi-level (e.g., participant- and study procedure-level) strategies that reduce off-schedule or missed assessments.