Researchers within the TTC Network have published multiple studies around TTC processes and outcomes, including the works previously cited7,8,9,10 as well as recent multi-year evaluations18, case study series19, and other analyses20. This work has generally been conducted within one of the three TTC Networks. However, possibly because funding for evaluation by any given TTC is contractually restricted, cross-network evaluation studies, including sociocentric network analyses of the type conducted in this study, have not previously been conducted.

The ONA component of our team’s evaluation affirmed that this approach to evaluating agency interactions and collaboration could be used to assess TA/TTC Network structures. ONA studies in public health are not especially rare21,22,23. However, findings from sociocentric analyses can sometimes be limited by the degree to which organizations choose to participate in data collection21. In contrast, SAMHSA’s TTC Network seemed particularly well-suited for sociocentric ONA, as the closed nature of the network (e.g., a specific, funded set of TTCs) and formal nature of the evaluation (e.g., responses from directors were expected by SAMHSA) further ensured the authors’ ability to collect a complete dataset. Similar analyses could likely be repeated in future evaluations and/or with subsequent TTC cohorts.

The ONA analyses also produced actionable information for SAMHSA to use in supporting its TTCs. First, the analyses showed fairly high levels of collaboration between TTCs within and across networks and regions. Importantly, the ONA allowed the evaluators to identify a particular region (Region F) where all three TTCs were relatively isolated for more than one interaction type, with ATTC F being isolated for three interaction types, and to recommend support and outreach to that region. Second, the premise of the NFACs is that they conduct cross-cutting work that is not limited to certain regions6, yet harmonic closeness data did not suggest that NFACs were consistently among the most centralized, although selected NFACs were among the more central TTCs for some interaction types. Those findings allowed the authors to suggest reexamination of the NFAC structure to learn why they were not more central to the collaboration and support networks and to build in additional qualitative evaluation components related to NFACs later in the overall evaluation. In both cases, ONA enabled potentially useful feedback to be delivered to SAMHSA that may not have been readily identifiable using other types of formative evaluation.

The most substantive limitation in interpreting these data is the low level of reciprocity in the collaboration networks. Reciprocity does not measure the actual collaborations between the TTCs, but rather the degree to which TTCs mutually asserted the same collaborations. If this discrepancy resulted from problems with memory or recall, then it may simply demonstrate data triangulation (e.g., multiple sources to identify single collaborations). On the other hand, if these differences resulted from different “thresholds” of what constitutes collaboration, then the analyses are more likely to reflect a “lowest threshold standard” (e.g., even “weak” collaborations are being documented). Furthermore, in the latter case, TTCs operating at the same recipient site might be more likely to indicate collaborations with each other. It is also theoretically possible that the reciprocity value for some dyads was suppressed in the specific case where a dyad of TTCs took the survey several weeks apart from one another, and their only interaction was in a period not covered by the overlapping retrospective timeframe provided in the question. While we cannot know whether that occurred, since all TTCs took the survey within the same 30-day period (and most within the same 21-day period), we view it as somewhat unlikely. Finally, it is also possible that the data were subject to social desirability bias.24 TTCs were aware that they were being evaluated and may have responded in a way that they thought SAMHSA or the evaluators would want to see. As with all studies, these outcomes and conclusions should be interpreted cautiously and not used to make broad inferences or decisions without consideration of the totality of available evidence.