The PACT study utilized a multiphase outcome evaluation incorporating a group-randomized experimental design testing the effectiveness of a novel intervention, the Program Sustainability Action Planning Model and Training Curricula, to increase the capacity for sustainability among state-level tobacco control programs. This study was approved by the Institutional Review Board of Washington University in St. Louis (reference number 201801196). This study also received approval under Washington University’s Protocol Review and Monitoring Committee. This study was also registered retrospectively on July 26, 2018, as a clinical trial (ClinicalTrials.gov/NCT03598114).

Intervention development and implementation

The primary goal of the PACT was to provide in-person, manualized training for sustainability action planning and assessment in public health programs. We used a multiphase approach over 5 years (2018–2023) to develop and implement an assessment of the effectiveness of the Program Sustainability Action Planning Model and Training Curricula. In the first phase of the PACT study, the intervention was developed through a rigorous multidisciplinary literature review process and a series of expert consultations. Using Kolb’s experiential learning theory [22], we developed the intervention to address the internal and external program-related domains proven to impact the capacity for sustainability of public health programs as outlined in the Program Sustainability Framework [3]. We used SCOPUS, ERIC (ProQuest), PubMed, Education Full Text, and PsychINFO databases to conduct formative reviews to inform the development and evaluation of the training intervention. Specifically, we performed literature reviews regarding experiential models of learning (i.e., duration and components) and technical assistance (type and duration) to design the intervention. To design the evaluation of the intervention, we conducted formative reviews to assess previous metrics of experiential learning and technical assistance effectiveness. Kolb’s model uses a four-step learning process: (1) concrete learning, (2) reflective observation, (3) abstract conceptualization, and (4) active experimentation [22]. Therefore, we designed a 2-day in-person action-oriented workshop that included (1) didactic presentation regarding program sustainability and the components of the Program Sustainability Framework (concrete learning), (2) discussion of current state program sustainability and state-specific challenges and facilitators (reflective observation), (3) exercises which helped state participants conceptualize and develop program sustainability objectives, and (4) development of a sustainability action plan to be implemented over 3 years. We also consulted with 2 academic experts in sustainability, 2 state tobacco control program directors, and 3 officials from the CDC Office on Smoking and Health to determine the final Program Sustainability Action Planning Model and Training Curricula [23]. In the second phase of this study, a multiyear, group-randomized trial was conducted to assess the effectiveness in improving the capacity for sustainability among state-level tobacco control programs (TCP). Ultimately, 11 intervention and 12 control TCPs participated. The Program Sustainability Action Planning Model and Training Curricula was delivered to 11 TCPs. The intervention consisted of a 2-day workshop to design a program sustainability action plan, 2 years of tailored technical assistance for implementing the action plan, and sustainability outcome assessment. Participants of the workshops actively engaged in developing state TCP-specific sustainability action plans. Each state action plan outlined 1 or 2 domain-focused objectives, matched with time-specific activities to be shared across stakeholders present. One person at each workshop claimed responsibility for overseeing the implementation process. Sustainability plans were designed to be implemented over the course of 2 years. All Program Sustainability Action Planning Training workshops followed the same structure but were tailored to each state depending on the Program Sustainability Framework domain chosen for the action plan. The 2-day workshop involved the TCP staff as well as a number of stakeholders (i.e., advocates, coalition members, voluntary organizations, grantees, local-level health department staff) actively participating to design a sustainability action plan and develop an implementation strategy. Inclusion of and participation by all stakeholders engaged were an important component of the sustainability action plan development process and ensuring all components of the state TCPs were considered through tailored workshops at baseline and ongoing, robust TA through their 3-year participation [23].

Our main hypotheses for the trial included the following:

1.

H1: Intervention group states will increase their capacity for sustainability more than the control group.

2.

H2: There will be a positive interaction effect between the group and the amount of dosage, meaning those in the intervention group will benefit more as the dosage increases.

3.

H3: The intervention will be more effective for states with lower policy progress (as proxied by the ALA smoke-free score) than those with higher policy progress.

Participating states and randomization

Our original sample consisted of the 50 US state tobacco control programs. A priori power analyses (at α = 0.05) revealed that between 9 (power = 0.8) and 12 (power = 0.9) states per group (control and intervention) would be appropriate. To randomize the two groups, we stratified the 50 states into four quadrants based on the states’ needs (as adult smoking rates) and tobacco control policy environments (as American Lung Association (ALA) smoke-free scores, 2015) [24]. The ALA score is a grade assigned to all 50 US states and the federal government that assesses the state of tobacco control on 4 key tobacco control policies: tobacco control and prevention spending, smoke-free air, tobacco taxes, and cessation coverage. In Fig. 1, smoking rates are on the x-axis, and ALA scores are on the y-axis. We created the quadrants using the mean scores (black horizontal and vertical lines). The state markers are sized by the percentage of CDC-recommended funding the states spend. We chose 3 states with different degrees of meeting the percentage of CDC-recommended funding [25] from each quadrant. We then chose the closest match (pair) for each state chosen based on the three characteristics displayed. Finally, we randomized states by pairs into the control or intervention group, to balance the 2 groups.

Fig. 1

Quadrant stratification for state selection

Measures

Data metrics were defined following recommendations from the advisory board and tobacco control experts and included organizational indicators, Program Sustainability Assessment Test (PSAT) scores, and intervention dosage. Organizational data was collected via record abstraction from annual state-level reports to the CDC Office of Smoking and Health. These reports address fulfillment criteria for the DP15-1509 funding announcement and describe the infrastructure, personnel, and activities of state tobacco control programs in detail. The funding announcements are a requirement of state programs, set by the CDC, to complete yearly reports of progress, goals, and challenges in order to receive federal funding. In addition to the CDC reports, other data was collected via secondary data sources, including the ALA’s annual State of Tobacco Control report [24] and the annual Healthy Americans report issued by Trust for America’s Health. The specific organizational metrics collected are described in a previously published manuscript [21].

In addition, we collected two primary sources of data. First, because it was not feasible to collect all data points through CDC program records, the study team developed a key informant interview tool to collect remaining programmatic information (e.g., staffing capacity and turnover, funding, and achievement of tobacco control goals). The interviews were conducted by phone interview with state program managers or other qualified surrogates (n = 21) and lasted 15–20 min. Responses were recorded, transcribed, and reviewed for completeness and accuracy. Any data remaining was collected via an online Qualtrics survey for the convenience of state program managers, who could not complete the phone interview. Only two state program managers or other qualified surrogates completed the online Qualitrics survey—all others (n = 21) completed the phone interview.

Data from the Program Sustainability Assessment Tool (PSAT) was also collected at 3 time points (baseline, 1 year post-intervention, and 2 years post-intervention). The PSAT consists of 40 7-point Likert scale items organized into the 8 domains of the program sustainability framework (environmental support, funding stability, partnerships, organizational capacity, program evaluation, program adaptation, communications, and strategic planning). The PSAT was emailed to all stakeholders who participated in the sustainability action planning process in each state. The range of participants per state was 7–15. To complete the PSAT, respondents rated the extent (1, little or no extent; 7, a very great extent) to which the program has or does what the item describes (e.g., “diverse community organizations are invested in the success of the program”). We calculated state-specific means for each of the 40 items. State-specific domain scores were obtained by averaging item scores within a domain. The overall domain scores were obtained by averaging the scores from all participating stakeholders for each domain, and standard deviations were calculated to show variability by state. These scores were used as the outcome in our analyses. The PSAT is a reliable instrument developed to evaluate the capacity for sustainability of public health, social service, and clinical care programs [2, 26].

Active dosage was measured in hours spent in sustainability training, technical assistance, or workshops delivered in-person or virtually. All programs (including control and intervention state TCPs) were given access to online sustainability resources (https://sustaintool.org/psat/resources/ and https://prcstl.wustl.edu/pact-resources/), referred to as passive dosage. A summary of intervention and control group activities can be seen in Table 1.

Table 1 Intervention and control group activitiesData and analyses

We tested these hypotheses using longitudinal linear mixed-effects modeling using data from the three time points annually in the intervention. We used random effects for the state and fixed effects for all other variables. The main predictors were group (hypothesis 1) and two types of dosage (active and passive). Active dosage was measured in contact hours spent in sustainability training, technical assistance, or workshops delivered in-person or virtually. Passive dosage was measured as a binary where 0 = no resource use and 1 = any resource use, as reported in annual surveys of programs. Other covariates included the percentage of CDC-recommended funding [27] (as a proxy for the level of program resources), ALA smoke-free score [28] (as a proxy for tobacco control policy progress), and program manager tenure as reported in annual surveys to represent program staff turnover or stability. This variable was measured categorically (vacant, less than 1 year, 1–3 years, 3–5 years, and more than 5 years in our annual surveys). In addition, we included interaction terms between the group and each type of dosage (hypothesis 2) and one for the group and ALA smoke-free score (hypothesis 3). The outcome variable was the annual PSAT score. For the model, we also tested for linearity in the model and normally distributed and independent errors. We used the R statistical environment for all analyses.