Eighty-two facilities from DoE Cohorts 1-5 collectively implemented 57 diverse practices. Twenty-five out of 57 practices (44%) were clinical interventions addressing a wide range of health care priorities, such as wound care, oral care, and medication safety. Twenty-three out of 57 practices (40%) were process improvements addressing diverse work related challenges (e.g., automated billing for home oxygen, SharePoint tool for construction safety, workflow management for artificial limbs). Nine out of 57 practices (16%) were staff interventions addressing various employee priorities, such as nursing stay interviews, new hire welcome program, and women’s health education. Fifty-three percent of practices (30/57) had a virtual component. Additional file 1 describes each of the 57 practices, which covered a wide range of VHA priorities around patient, staff, and health system needs.

As of 2021 (timepoints ranged from 1-5 years after the 6-9-month facilitated implementation support period), about one-third of facility representatives reported their practice was fully sustained, one-third reported their practice was not fully sustained, and one-third did not respond. One facility representative was missing outcomes for all time-points. Thirty percent (25/82) of facility representatives reported consistent (the same) outcomes from initial implementation to sustainment and 70% (57/82) of facility representatives reported inconsistent (changing) outcomes from implementation to sustainment.

Figure 1 visually displays all longitudinal pathways, showing shifts in practice implementation/sustainment status over time. The main facilitating and hindering factors influencing sustainment, identified from directed content analysis of open-ended text boxes, are noted in italics below. Additionally, Fig. 2 visually compares these key factors that facilitate and hinder sustainment by outcomes. Tables 3 and 4 list all factors that facilitate and hinder sustainment for the 66% (33/59) of facility representatives who responded to optional open-ended text boxes. Exemplar quotes are provided in Tables 3 and 4 and Figs. 4, 5, 7 and 8. Additional file 3 provides more details, showing sustainment outcomes separately by practice type.

Fig. 1

Pathways for all facilities. *Implementation follow-up was only provided for facilities that did not fully implement their practice or were missing data. **Cohort 4 practices only have sustainment outcomes for 2020 and 2021. Cohort 5 practices only have sustainment outcomes for 2021. For the purposes of the figure visual, their statuses were carried over from earlier timepoints

Fig. 2

Venn diagram comparing factors that facilitate and hinder sustainment

Table 3 Factors that facilitate sustainmentTable 4 Factors that hinder sustainmentLongitudinal pathways for practices that were fully sustained in 2021

Thirty-five percent (29/82) of facility representatives reported their practice was fully sustained in 2021, which was an average of 2.3 years (range: 1-5 years) after implementation. Of these 29 facility representatives, 76% (22/29) reported full implementation after their 6-9-months of facilitated implementation support period; the remaining facility representatives completed their implementation milestone later. Further, of these 29 who reported their practice was fully sustained, 79% (23/29) reported full sustainment at initial follow-up, which was approximately one year after the implementation period. Whereas 21% (6/29) of facility representatives who did not initially sustain their practice went on to sustain their practice by 2021.

Facilitators of sustainment included: demonstration of practice effectiveness/benefit, sufficient organizational leadership, appropriate workforce, and practice adaptation/alignment. Facility representatives also described potential barriers to future sustainment, including workforce turnover, challenges with critical incidents related to the COVID-19 pandemic, and concerns about ongoing support; they also listed potential facilitators for future sustainment that included having appropriate workforce and sufficient organizational leadership and ongoing support. See Tables 3 and 4 for all factors influencing sustainment. Sustained practices included more clinical interventions (45%, 13/29) and process improvements (45%, 13/29) compared to staff interventions (10%, 3/29) and were almost evenly split between the presence (48%, 14/29) or absence (52%, 15/29) of a virtual component. Figure 3 displays longitudinal pathways for facility representatives that fully sustained their practice.

Fig. 3

Pathways for facilities that fully sustained by 2021. *Implementation follow-up was only provided for facilities that did not fully implement their practice or were missing data. **Cohort 4 practices only have sustainment outcomes for 2020 and 2021. Cohort 5 practices only have sustainment outcomes for 2021. For the purposes of the figure visual, their statuses were carried over from earlier timepoints

Consistent pathways

Among the 29 facility representatives who reported their practice was sustained in 2021, 66% (19/29) consistently sustained; meaning that they had sustained at all timepoints after fully implementing. All facility representatives anticipated that their practice would continue to be sustained into the future and 89% (17/19) reported their practice was institutionalized and effective. See Fig. 4 for a consistently successful pathway showing outcomes with qualitative explanations.

Fig. 4

Facility with a consistent pathway to full sustainment: example outcomes and qualitative explanations

Inconsistent pathways

Among the 34% (10/29) of facility representatives with an inconsistent pathway to sustainment in 2021, the majority experienced initial challenges with implementation (60%, 6/10) compared to challenges with implementation and sustainment (10%, 1/10) or were missing data (30%, 3/10) in 2020. Facility representatives with inconsistent pathways to sustainment needed more calendar time than the 6-9-month facilitated implementation support period to overcome implementation barriers, which were often related to insufficient workforce and available resources. One facility representative also reported a temporary pause in sustainment due to critical incidents related to the COVID-19 pandemic that were resolved when pandemic restrictions at their facility were loosened.

Once achieving sustainment, these facility representatives emphasized that facilitators to sustainment were sufficient workforce and organizational leadership. All but two facility representatives anticipated future sustainment. One facility representative anticipated future sustainment as “unlikely” without explanation and one did not respond to the question (see Fig. 3). Most (80%, 8/10) facility representatives described their practice as institutionalized and effective in 2021. Whereas, fewer (20%, 2/10) facility representatives described partial institutionalization because of no/limited funding or lack of adequate number of service users (i.e., insufficient Veteran enrollment in a voluntary program). Two out of ten (20%) facility representatives reported their practice was partially effective: one of whom cited no/limited funding as an issue. Figure 5 provides an example an inconsistent but successful pathway showing outcomes and qualitative explanations.

Fig. 5

Facility with an inconsistent pathway to full sustainment: example outcomes and qualitative explanations

Longitudinal pathways for practices that were not fully sustained in 2021

Thirty-seven percent (30/82) of facility representatives reported that their practice was not fully sustained because they were in a “liminal” stage [31, 32] (neither sustained nor discontinued) or permanently discontinued as of 2021, which was an average of 2.1 years (range: 1-5 years) after implementation. Only 43% (13/30) of these facility representatives reported full implementation after 6-9-months of facilitated implementation support with five additional facility representatives completing implementation later. Only 23% (7/30) of facility representatives reported full sustainment at initial follow-up, which was approximately one year after the implementation period.

Barriers to sustainment included: insufficient workforce (losing or not being able to hire staff), not being able to maintain EIP fidelity/integrity, critical incidents related to the COVID-19 pandemic, organizational leadership did not support sustainment of EIP, no ongoing support, lack of trained personnel to continue the EIP, and/or EIP effectiveness/benefit was not observed. Despite not being fully sustained, these facility representatives also described facilitators to sustainment. The most frequently reported facilitators to sustainment were internal/external EIP champions and sustained/attention to topic/priority, which were not mentioned by facility representatives with sustained practices. See Tables 3 and 4 for all factors influencing sustainment. Practices that were not fully sustained had a similar percent of clinical interventions (50%, 15/30), process improvements (43%, 13/30), and staff-oriented interventions (7%, 2/30) as those that were sustained but had fewer practices with virtual components (37%, 11/30). The following sections describe results for facility representatives who reported un-sustained practices, which is organized by status (“liminal” sustainment or discontinued permanently) and pathway (consistent or inconsistent). Figure 6 displays longitudinal pathways for facility representatives that did not fully sustain their practice.

Fig. 6

Pathways for facilities that did not fully sustain by 2021. *Implementation follow-up was only provided for facilities that did not fully implement their practice or were missing data. **Cohort 4 practices only have sustainment outcomes for 2020 and 2021. Cohort 5 practices only have sustainment outcomes for 2021. For the purposes of the figure visual, their statuses were carried over from earlier timepoints

Liminal sustainment

Eighteen percent (15/82) of facility representatives reported that their practices were not fully sustained because they were in a “liminal” stage of sustainment (40%, 6/15 partially sustained; 60%, 9/15 temporarily paused) since they were neither sustained nor discontinued in 2021. The major barriers associated with practices that were in a “liminal” stage of sustainment included insufficient workforce, no ongoing support, lack of trained personnel, and critical incidents related to the COVID-19 pandemic. Though fewer facilitators were mentioned compared to barriers, the top facilitator to sustainment was internal/external EIP champions.

Despite their “liminal” status, almost half (7/15) of these facility representatives were optimistic about sustaining their practice in the future. However, the remaining facility representatives did not expect to sustain their practice (33%, 5/15) or were uncertain about future sustainment (20%, 3/15). Most of these facility representatives reported their practice was not fully institutionalized (53%, 8/15 no; 27%, 4/15 partial) nor effective (20%, 3/15 no; 53%, 8/15 partial).

Permanently discontinued

Eighteen percent (15/82) of facility representatives reported that their practices were not fully sustained because they were permanently discontinued. Common barriers associated with practices that were discontinued included two of the same as those in a “liminal” stage (workforce and critical incidents). However, not able to maintain EIP fidelity/integrity (top barrier), organizational leadership did not support sustainment of the EIP, and system policy change were cited as other important reasons for practice discontinuation. Despite practices being discontinued, two facility representatives cited sustained attention to topic/priority as a facilitator.

Among facility representatives who provided responses to secondary outcomes (40%, 6/15), fifty percent (3/6) reported their practice was not fully institutionalized (33%, 2/6 no; 17%, 1/6 partial). Unexpectedly, the remaining facility representatives with discontinued practices (50%, 3/6) reported their practice was institutionalized due to some aspect of the practice becoming routinized. Regarding effectiveness, more facility representatives reported their practice was not fully effective (50%, 3/6 no; 17%, 1/6 partial). However, those who reported their discontinued practice had demonstrated effectiveness (33%, 2/6), cited they were tracking an aspect of practice effectiveness (e.g., continued using charts to show progress).

Consistent pathways

Only 17% (5/30) of facility representatives reported consistently less successful implementation and sustainment outcomes over time. Two out five (40%) facility representatives reported their practice being partially implemented and sustained through 2021. Despite these facility representatives’ consistent “liminal” status, responses to secondary outcomes of institutionalization, effectiveness, and anticipated sustainment were different from each other. One out of these two (50%) facility representatives reported partial institutionalization and effectiveness and anticipated full sustainment in the future, but it was dependent on having sufficient workforce in place. The other facility representative (1/2, 50%) reported that the practice was effective but was not institutionalized and would not be sustained in the future due to insufficient workforce.

The remaining three out of five (60%) facility representatives who were consistently less successful did not implement their practice by the end of the 6-9-months of facilitated implementation support and then reported their practice was permanently discontinued. Only 1/3 (33%) facility representatives responded to the institutionalization and effectiveness outcome questions and responded that their practice was not institutionalized nor effective. These 3/5 (60%) facility representatives experienced insurmountable barriers with implementation and never reached the sustainment phase because of critical incidents related to the COVID-19 pandemic or no/limited funding. Figure 7 provides an example pathway of a consistently unsuccessful facility showing outcomes and qualitative explanations.

Fig. 7

Facility with a consistent pathway to not fully sustained: example outcomes and qualitative explanations

Inconsistent pathways

Eighty-three percent (25/30) of facility representatives whose practices were not fully sustained in 2021 reported inconsistent outcomes over time, which meant their outcomes did not align over time and/or they were missing at least one outcome prior to 2021. There were two main types of inconsistent pathways leading to unsuccessful sustainment. The first type consisted of facility representatives who successfully implemented their practice, but experienced challenges with sustainment. Early on, 13/25 (52%) facility representatives reported full implementation at the end of the 6-9-months of facilitated implementation support period and another 5/25 (20%) reported full implementation with additional time when responding to the follow-up survey. Although 72% (18/25) of these facility representatives were successful at implementation, by 2021, 50% (9/18) downgraded to a “liminal” stage of sustainment (22%, 4/18 partially sustained; 28%, 5/18 temporarily not sustained) and 50% (9/18) reported being discontinued permanently.

The second type consisted of facility representatives (28%, 7/25) who experienced some challenges early on with implementation (42%, 3/7 not implemented; 29%, 2/7 partially implemented) or were missing implementation outcomes (29%, 2/7). These facility representatives’ status fluctuated up and down over time and by 2021, they all downgraded to being temporarily not sustained (57%, 4/7) or permanently discontinued (43%, 3/7). Figure 8 provides an example of an inconsistent not fully sustained pathway showing outcomes and qualitative explanations.

Fig. 8

Facility with an inconsistent pathway to not fully sustained: example outcomes and qualitative explanations

Missing data in 2021

In 2021, 28% of facility representatives (23/82) were missing their most recent sustainment outcome. Only one out of 23 facility representatives (4%) had consistently missing implementation/sustainment outcomes over time compared to 22 (96%) facility representatives who had previously provided outcomes (i.e., inconsistent outcomes pathway due to changes over time). Of the 23 facility representatives lost-to-follow-up in 2021, only 2/23 (9%) were lost-to follow-up two years earlier in 2019. However, by 2020, an additional 12/23 facility representatives (52%) were lost to follow-up, and the remainder (9/23; 39%) had their first missing data in 2021. Facility representatives with missing 2021 sustainment outcomes had more process improvement practices (39%, 9/23) compared to staff interventions (35%, 8/23) and clinical interventions (26%, 6/23), which differed from practices that were sustained or not fully sustained. These facility representatives also had fewer practices with virtual components (30%, 7/23), which was like those that were not fully sustained.

After the 6-9-months of facilitated implementation support period, only 17% (4/23) of facility representatives had missing implementation outcomes data. However, three out of four (75%) facility representatives responded at follow-up. One of these three facility representatives reported their practice was implemented and sustained before being missing in 2021. Whereas the other two other facility representatives reported at follow-up that their practice was not implemented nor sustained before being missing in 2021.

Seventy-percent (16/23) of facility representatives with data missing in 2021 reported they fully implemented by the time of the second implementation assessment. Although these facility representatives were lost-to-follow-up in 2021, most (68%, 11/16) reported full implementation or sustainment as their last known status. The remaining five facility representatives reported a downgraded status of being temporarily (13%, 2/16), partially (6%, 1/16), or not (13%, 2/16) sustained before being lost-to-follow-up in subsequent assessments. See Fig. 9 for pathways of facilities with missing outcomes in 2021.

Fig. 9

Pathways for facilities with missing data in 2021. *Implementation follow-up was only provided for facilities that did not fully implement their practice or were missing data. **Cohort 4 practices only have sustainment outcomes for 2020 and 2021. Cohort 5 practices only have sustainment outcomes for 2021. For the purposes of the figure visual, their statuses were carried over from earlier timepoints

Missing data overall

With respect to missing data trends across all 82 facility representatives, 41% (34/82) percent had at least one missing time point across all years of data collection with an average of 1.8 missing time points overall. Only 1/82 (1%) facility representative was consistently missing outcomes for all time-points. Among the 5/82 (6%) facility representatives with two missing timepoints in a row, only 40% (2/5) responded to subsequent surveys. Facility representatives from Cohort 1 (53%, 9/17) and Cohort 3 (43%, 6/14) had more missing data than those from other Cohorts (Cohort 2 = 24%, 4/17; Cohort 4 = 19%, 3/16; Cohort 5 = 17%, 3/18).