Twenty four colleges with student counseling centers were cluster randomized to the two implementation strategies, matched on the ratio of the number of students per site divided by the number of therapists per site; details regarding the study, methods, and therapist outcomes are published elsewhere [2, 8]. To summarize, at colleges assigned to the experimental (i.e., TTT) condition, one therapist was selected as the trainer by the director of the study at the counseling center. This trainer attended two workshops – the first 2-day workshop trained participants in how to deliver IPT. Following this workshop, the trainer returned to their counseling center and treated up to two cases with IPT; this clinical practice was supervised, and feedback was provided to help the trainer improve. The second workshop provided the trainer with the tools to teach other clinicians how to deliver IPT, and to ensure quality control. Once the trainer had completed both workshops, they began to train their colleagues in IPT. This phase of training lasted approximately six months. Trainers met weekly with their trainees for optional group consultations and participated in optional monthly phone calls with the research team member who conducted their training and with their peer trainers from other sites.

At colleges assigned to the control (i.e., expert consultation) condition, therapists participated in a 2-day IPT workshop identical in content and structure to the first workshop delivered to therapists in the TTT condition (above). Following training, the research team member who conducted the workshop continued to engage with their trainee therapists in an optional hour-long phone call each month for up to a year in order to support them in implementing IPT.

A total of 184 professionals—95 in the TTT condition and 89 in the expert consultation condition—formed the initial cohort of therapists at participating counseling centers. Attrition of therapists occurred for reasons such as leaving the site to seek another opportunity, withdrawal of consent, retirement, and no longer seeing patients (further details regarding therapist inclusion and exclusion criteria, and the CONSORT diagram, are published elsewhere [2]. At study end, 60 therapists in the TTT condition and 55 therapists in the expert consultation condition – all of whom had audio-recorded their sessions – were included in the analysis.

The primary outcome measure was the change in therapist fidelity, an implementation outcome [6], which was obtained from the parent study [2, 8]. Fidelity was operationalized into two dimensions of adherence and competence, and each was assessed during two assessment timeframes (i.e., at baseline and after training). Baseline assessments were conducted before any training had occurred and captured whether therapists had prior experience with IPT. Post-training assessments were conducted after completion of all relevant workshops, within a 6–12 month window following recruitment, depending upon condition. These two dimensions were assessed from audio recordings of therapy sessions, to which therapists had consented, and of which they were aware. Assessments were conducted by raters who were masked to the implementation strategy and used the IPT Fidelity Rating Scale to determine fidelity [16]. Adherence scores ranged from 0 to 1. Competence scores ranged from 0 to 2. Raters included a senior study team member and five graduate student research assistants who were masked to implementation condition. Interrater reliability, as calculated from a sample of 9 audio recordings after training, was reported as 0.72 (95% CI, 0.46–0.91) [2].

We obtained information on changes in fidelity scores directly from the parent study [2, 8]. The parent study reported outcomes from standard linear mixed effects models conducted in Mplus version 8 [17]. Therapist outcomes were based on intent-to-treat; outcomes from all randomized therapists were included as long as a therapist had at least one post-training assessment. Models were specified as random intercept models, all standard errors were estimated using robust maximum likelihood, and effect sizes for adherence and competence outcomes were reported as Cohen’s d. Changes in adherence and competence were reported as b, which is a nonstandardized coefficient obtained from regressing the slope on centered training condition, centered baseline covariates, and their interactions. All outcomes were compared both within and across conditions.

We gathered costs of the two implementation strategies using three methods, as described below. First, we developed and fielded an implementation cost survey, which was administered before implementation activities came to an end at the site. Second, we conducted telephone follow up interviews at sites to validate and clarify survey responses. Finally, we obtained time tracking logs maintained by the research team that trained therapists across both conditions.

We used the Survey Monkey platform (www.surveymonkey.com) to deploy the implementation cost survey designed to capture labor and nonlabor costs associated with implementation. (Supplement 1 lists the key labor and nonlabor cost items gathered in the TTT condition; cost items for the control condition are a subset of those in the TTT condition.) The survey was disseminated prior to the end of all study activities at the site, was completed by the study director at each site, and was based on an time-driven activity-based costing approach [18, 19]. Items elicited details about the staff engaged in the study, salaries for therapists, amounts of time spent by therapists in each part of the implementation process, and nonlabor costs such as facility and equipment costs.

Prior pilot work on costing had revealed that many mental health clinician/administrator respondents were unused to thinking about their work in terms of costs. Hence, we conducted an hour-long, semi-structured, follow-up interview with the study director at the site, scheduled as soon as feasible following receipt of the cost survey. This interview went over the individual items listed in Supplement 1, ensuring that respondents had understood the questions, clarifying any items that were unclear, and reconfirming responses. Directors reviewed the survey items they had previously completed, consulted additional documents (if needed), and were asked to procure any other information, in order to enhance the validity of their responses. All directors participated in the interviews. Two members of the study team jointly conducted the first 4 interviews to enhance reliability.

Third, we independently attempted to increase the validity of therapist self-report using tracking logs maintained by the research team. Tracking logs are contact logs or time logs that capture the amount of time spent in specified activities. The expert maintained a tracking log of his time as he undertook training activities across both conditions. In the expert consultation (control) condition, therapists self-reported the time that they spent consulting with the expert. This self-reported time was compared with the time log of the expert who provided such consultation. In the TTT condition, trainers self-reported their time spent consulting with the expert, which was compared with the expert’s time log. However, in this condition, the training of individual therapists is overseen by the trainer within the site, not by the expert, and so we have no way to validate the time spent by the trainer in training therapists. Because we were able to validate only part of the overall labor costs in the TTT condition, we opted to rely on self-reported times, available across both conditions, in this study.

We obtained labor and nonlabor costs associated with implementing IPT. Labor costs are generated by trainers, therapists, research staff, and administrators located in participating counseling centers, as a result of performing activities associated with implementation; examples include the cost of practitioner time spent in training, and the time costs of trainers engaged in supervision and training in the TTT condition. These costs, listed in Supplement 1, were aggregated at the level of a counseling center. Nonlabor costs were direct monetary costs to centers of implementing the two conditions, and include the cost of materials and manuals, travel, and supplies. Clinicians at college counseling centers were not required to meet defined productivity metrics in terms of hours spent seeing clients. Counseling centers operated on university-established budgets and did not directly bill students or insurers for services. Our perspective is solely that of the counseling center, not of the parent university or other payors.

All 24 sites submitted surveys and participated in interviews. In the expert consultation condition, we re-interviewed 3 therapists who had reported unusually high in-house consultation hours and corrected the numbers. Missing time values of all group activities (attending expert consultation, time spent in in-house peer consultation) were filled based on mean self-reported times of other clinicians attending the same groups at the same time at the same site (i.e., group mean substitution). For individual activities (time spent in reading materials before and after training), we used mean self-reported times from all therapists in that condition (i.e., site-level mean substitution).

We entered data from the survey into a worksheet detailing time spent by individual therapists, clustered by site, on all implementation activities in order to first produce a per-therapist cost of implementation. Because of confidentiality concerns with individual salaries, time spent by therapists in implementation activities was multiplied by the site average for salary and fringe to arrive at a per therapist cost. Therapist costs were aggregated at the center level, to which we added all additional costs incurred by the center for training activities (photocopying, telephone charges, etc.). Labor and nonlabor costs of the expert were also charged to the center and added. Here, we report costs per study therapist trained, averaged across all centers assigned to each condition.

We calculated the difference in the mean implementation costs of TTT versus expert consultation incurred by a counseling center without discounting, given the time horizon of the study. We then divide that by (a) the difference in mean adherence scores, and (b) the difference in mean competence outcomes of therapists within that center for each implementation strategy in two separate calculations, one for each outcome. We did not examine clinical outcomes of the students (clients) served by therapists at these counseling centers.

Quantifying Thresholds for Therapist Training Outcomes.

We were unsuccessful in arriving at an a priori willingness to pay threshold to inform the cost-effectiveness analyses. Consequently, we followed the suggestions of Briggs and colleagues [20], and present uncertainty around our estimates. We estimated the joint density of variations in the incremental costs of training therapists incurred by counseling centers (i.e., difference in the mean per-therapist training cost between the conditions) and incremental fidelity outcomes (difference in outcomes between conditions) for statistically significant relationships. We ran 1000 replications of a non-parametric bootstrap re-sampling of costs (from the current study) and outcomes (from the parent trial), to populate a scatter plot.

We conducted a total of 5 sensitivity analyses – (i) As described below, counseling centers in our study trained both therapists that participated in the study (“study therapists”), as well as those that were not part of the study (“non-study therapists”). In order to quantify economies of scale among sites that trained a large number of non-study therapists, we quantified the mean per-therapist cost of training study therapists only, and the mean per-therapist cost of training all therapists regardless of participation in the study, for each center that trained non-study therapists. We then differenced these to obtain cost savings on a per-therapist basis for the sites that also trained non-study therapists. (ii) In order to test if it is more cost-effective to train highly skilled therapists as trainers, we used a design feature of the parent study, in which trainers in the TTT condition received training and supervision until they were deemed ready by the expert to train others in IPT. We calculated mean therapist costs for variations in training intensity for trainers – at the 10th percentile of post-training consultation and supervision time (suggesting that these trainers were highly skilled and did not need a great deal of training) versus the 90th percentile of training time. (iii) As we describe below, between a quarter and a third of therapists across various sites possessed prior experience in the use of IPT [2]. We calculated mean per-therapist costs for those therapists whose baseline adherence and competence scores were above the mean baseline score for all therapists in those respective conditions (indicating that these were therapists skilled in IPT even prior to training). We compared these training costs to the costs of training therapists whose baseline adherence and competence scores were below the mean. (iv) We quantified the cost implications of using therapist self-report versus expert-report when it came to quantifying training time. We did this for the expert consultation condition alone because the time taken for each task in this condition is independently quantified by both therapists and the expert. (v) We conducted sensitivity analysis for missingness in the time taken for the performance of individual tasks by therapists in the TTT condition. We used the 10th and 90th percentiles of self-reported times as our range of plausible values, and then estimated costs at these two values for all therapists in order to obtain a range.