Study aims and hypothesisImplementation component

Aim 1: Determine whether implementation of the Navicare model sustains access (co-primary outcome) to mental health services in rural and remote settings in the post-implementation period compared to the implementation period.

Aim 2: Determine the success (or otherwise) of implementing, scaling up and sustaining Navicare in four regional communities, and likely mechanisms of action.

Aim 3: Understand the core and adaptable components of the Navicare model and implementation strategies across the study and within four communities in the Greater Whitsunday Region, Queensland.

Aim 4: Provide conceptual evidence of the strengths and limitations of the EPIS framework when applied to rural and remote mental health in Australia and suggest refinement, complementary or alternate approaches for future studies.

Hypothesis: The implementation of Navicare will sustain access (co-primary outcome) to mental health services in local rural and remote settings in the post-implementation period compared to the implementation period.

Effectiveness component Cost-consequences component Context

The study will implement and evaluate implementation of Navicare at an existing site and three new sites in the Greater Whitsunday Region in rural and remote Queensland. The Region has a population of 186,512 people and covers 90,354 square kilometres across several local government areas [17]. New sites that are the focus of the effectiveness component of the study will be determined as part of the study process based on community consultations, along with assessment of contextual factors and site readiness.

Study design

A pre-post comparison, community-engaged hybrid type 2 effectiveness-implementation study design will focus on the implementation of Navicare at three new sites as well as monitoring implementation at an existing site established since 2021. The rationale for choosing a type 2 design aligns with criteria reported for these designs [18]. This includes having some but not strong evidence of positive effects of components of the Navicare model on mental health, the likely need for adaptation of the intervention across sites, some existing evidence regarding barriers and facilitators to implementation, and momentum to evaluate Navicare to respond to urgent community needs whilst being implemented. A stepped wedge design with staggered multiple baseline implementation combines repeated measurement and analysis of outcomes within community sites with those captured over time across all sites [18] (Fig. 2). This design is well-suited to pragmatic evaluation of whole-of-community interventions where implementation is likely to do more good than harm, and withholding the intervention via randomisation would be seen as ethically, socially or practically unacceptable [19]. We considered this design to be ideal for evaluating the implementation and effect of Navicare across interconnected communities and where uptake of service access may increase over time.

Fig. 2

Timeline for each EPIS phase at each site

A community-engaged implementation approach includes community-academic partnerships and community-based participatory research methods [20, 21]. Adopting such an approach was seen as a critical component of implementation success and sustainability by participants in our pilot work in the Region [9]. This community-engaged approach adopts the principles of iterative engagement of diverse stakeholders through all study phases, partnering in implementation decision making and planning, valuing community strengths, tailoring to local context, evaluating meaningful outcomes, and using pragmatic flexible implementation approaches. Besides improving study execution (including implementation, participant recruitment, and data collection), participatory methods can also improve mental health outcomes, collaboration between diverse stakeholders and researchers, and lead to system change [22].

Implementation outcomes and potential mechanisms that influence the study outcomes (including community readiness) will be the focus of this evaluation. A pre-post effectiveness study will use routinely collected data gathered before, during and after Navicare implementation to determine effectiveness of the Navicare model on health service and client outcomes in the region at a population level. Additionally, a longitudinal contextual assessment across the study period will dynamically map the contextual factors influencing Navicare delivery, the need for adaptation and sustainability. It is expected that this evaluation will lead to further refinement or suggestions for refining Navicare including recommendations for implementation strategies that pertain to other rural and remote sites in scaling up the model beyond the study.

Theoretical approach

Five implementation science frameworks underpin the study.

Exploration, preparation, implementation and sustainment (EPIS) framework

EPIS was selected as it was developed based on literature about the implementation of innovations in public sector social and allied health service systems including mental health in the United States [23]. The framework has been applied extensively in mental health sectors, including in Australia [23], as well as in cancer control in sub-Saharan Africa [24]. It also has a considerable focus on factors in the outer context (e.g., health system), and bridging factors that interconnect the inner and outer context, both of which were identified as critically important determinants in our initial research to develop the intervention with community members [9]. The influence of other EPIS domains, the inner context and the innovation being implemented [16] will also be examined. EPIS is particularly well suited to examining the role of implementation power, equity and person-oriented recovery [8]—often neglected factors in evaluating the success of mental health interventions [25].

In recognition of the importance of the exploration and preparation phases of EPIS in the initial co-design work [9] and to the success of innovations more broadly [26], our study design includes these phases prior to the implementation of Navicare at any new site. As a determinants and process framework, use of EPIS will allow for a prospective understanding of determinants and mechanisms of implementation, and tailoring of implementation strategies to overcome these determinants mapped to the EPIS phases [26], scaling-up across multiple sites, and evaluating interconnections and variance across factors and phases [8]. The economic evaluation, engagement with stakeholders, and adaptation of the implementation to the local context at each site will also be mapped to the EPIS phases to determine the activities, costs and extent of engagement and adaptation in each phase and across phases.

All four EPIS phases and activities will be sequentially applied at the three new study sites. New sites will be identified through consultation with key stakeholders as part of our community-engaged approach in the exploration phase. The proposed timeline for each phase at each site is outlined in Fig. 2, however we expect the duration of time spent in the exploration and preparation phases may change based on circumstances at the local sites. Study activities by EPIS phase for each site are outlined in Fig. 3, including community consultations, workshops, and site implementation stakeholder groups. The exploration and preparation phases have already been completed at the existing site where implementation commenced in 2021. For this existing site, evaluation will be conducted in the implementation and sustainment phases only. Study activities in the exploration, preparation and implementation phases will be the same across all new sites (Figs. 2 and 3). This is consistent with EPIS, in recognising that the effectiveness of implementation is influenced at least in part by activities in the exploration and preparation phases [26]. Implementation strategies, however, may be adapted and tailored for sites based on the localised contextual factors identified in the Exploration and Preparation phases at each site.

Fig. 3

Study activities by EPIS phase for each site and scaling-up

Implementation Outcomes Framework (IOF)

Proctor’s framework was selected to guide selection of the implementation outcomes for this project and includes feasibility, acceptability, appropriateness, adoption, cost of implementation, fidelity and sustainability [27, 28]. Definitions of each of the constructs along with their operationalisation in the study are outlined in Table 1.

Table 1 Implementation outcomes: description, data type, source and timepoint of measurementFramework for Reporting Adaptations and Modifications – Enhanced (FRAME) and Evidence-based Implementation Strategies (FRAME-IS)

FRAME consists of eight aspects including whether the intervention adaptation was planned or unplanned, the extent the adaptation is fidelity consistent and the intent or goal of the modification [30]. The FRAME-IS consists of four core elements to guide documentation: (1) describing the evidence-based practice, implementation strategy and adaptation; (2) what is adapted; (3) the nature of the adaptation; and (4) the rationale for the adaptation [31]. Delivery and adaptation of the Navicare intervention and implementation strategies will be guided by FRAME and FRAME-IS respectively across the study period and sites. This will assist in determining modifications to Navicare and implementation strategies that were associated with successful versus unsuccessful implementation as well as core implementation strategies [30, 31].

World Health Organisation ExpandNet framework (ExpandNet/WHO)

The validated ExpandNet/WHO framework consists of the elements of the intervention, user organisation(s), environment, resource team, and scaling-up strategy and is guided by the principles of systems thinking, a focus on sustainability, the need to determine scalability, and respect for gender, equity and human rights principles [32, 33]. The framework will be utilised in developing a scalability strategy with stakeholders during a roundtable workshop.

Implementation study component methods

This study component will address aims 1 to 4.

Population

The inclusion criteria for service user or carer participants (over the age of 16 years) are, being a current or past user of Navicare in any of the site locations or broader mental health services in the region. Service providers and other participants will be included if they are a current or past service provider through Navicare or are connected with Navicare as a relevant stakeholder, including Care Navigators, community members, government agencies and policy makers. Roundtable stakeholder participants will include: experts in mental health across Australia (including those in regional mental health and recovery-based programs); at least two people with lived experience of mental ill-health; a Care Navigator from Navicare; and representatives from non-government and government mental health agencies, including policy makers.

Recruitment

Potential individual participants (e.g., service users, carers and service providers) will be contacted by a member of the study team or a study partner with an existing relationship to the provider or user, but not by the service providers of users (e.g., psychologists, social workers). Initial contact will be made in the form of an email, newsletter or flyer outlining the intent of the research and information on how to participate. Potential participants who are interested in taking part will be required to contact the research team if they would like to participate, following which information and consent forms will be sent to them. Potential participants who respond to a request to participate may be asked to complete screening questions as part of the process to ensure diverse representation in qualitative study activities (e.g., age, gender). Participants must provide written consent prior to engaging in community consultations, interviews, implementation diaries, the roundtable, and workshops. Non-English speaking potential participants and people with language impairments will be able to participate if they have a support person who is able to assist with translation and language assistance.

Data collection

The sampling approach for qualitative data collection will be convenience sampling initially, followed by purposeful sampling then theoretical sampling [34] with diverse representation sought as part of our recruitment strategy (Table 2). Routinely collected deidentified Navicare service-level data will be accessed, including sociodemographic data and telehealth usage. Sociodemographic data will also be collected from Navicare users and providers who consent to participate in interviews, focus groups and surveys.

Table 2 Characteristics of stakeholder groups sought for diverse representationCo-primary outcome

Our novel co-primary outcome measure of sustained access to Navicare was identified in our initial intervention co-design work with communities as being of great importance [9]. This outcome will be evaluated using the total number of new eligible persons with or at risk of mental ill-health or their carers who seek assistance from or through Navicare each month either directly or by provider referral. These data will be obtained from routinely collected Navicare service data. Access will be measured monthly across the study, comparing implementation (12-months) and sustainment (12-months) phases across all new sites, with commencement defined according to when the first client is seen physically at each new local site (see Fig. 2). We consider this outcome to bear similarity to Proctor’s outcome of equitable access, with access hypothesised to combine the outcomes of penetration, sustainability and fidelity [28].

Implementation outcomes and adaptation

Mixed methods data will contribute to the outcomes of acceptability, adoption, fidelity, sustainability, cost of implementation, implementation, and adaptation. Qualitative data will contribute to feasibility, appropriateness and scaling-up. The following outcome measures and approaches will be used, with measurement timepoints and further details outlined in Tables 1 and 3. Outcome measures to be used include the Navigation Satisfaction Tool (NAVSAT) [35], Program Sustainability Assessment Tool (PSAT) [36] and a fidelity-adaptation tool to be developed as part of the study. The tool will document core non-adaptable and modifiable intervention components and the extent to which Navicare intervention activities are delivered as planned or adapted to meet local community needs, and explore reasons underpinning adaptations (detailed in Table 3).

Table 3 Protocol outcome measures

Implementation diaries will be used to collect information on everyday work of Navicare to report on the process of care navigation as well as being a tool to capture resource use, and barriers and facilitators to implementation. Implementation diaries will be completed by Care Navigators throughout the implementation and sustainment phases to allow an in-depth understanding of project implementation by capturing project evolution and real-time insights while reducing self-reporting bias [39, 40]. The structure of the diaries will be based on the EPIS framework constructs as well as implementation outcomes relevant to the study. Implementation diaries will be completed monthly in the implementation and sustainment period at the new study sites, hosted in Redcap, an electronic data capture tool [41].

Context assessment and evaluation of contextual factors (for example, barriers and facilitators, mechanisms) will be conducted using a hybrid inductive-deductive approach. Constructivist methodology [42, 43] will underpin the inductive approach, and EPIS study phases will inform the deductive approach to capture how these factors influence the findings. Using constructivist methodology has been reported to promote a multi-layered approach and can assist in building a holistic understanding of context as dynamic across the study rather than static, capturing interactions within context and maximising uptake and sustainability when used to inform subsequent phases of implementation [42]. Context and interactions between context, intervention, and implementation will be used to prospectively inform subsequent study phases and will be analysed separately from implementation processes [42]. Data sources will include semi-structured interviews, community consultations, and workshops with service users, carers, and service providers at all sites; and observations at forums or site visits. Field notes and memos will be made in relation to all data sources.

Scaling up nationally will be prepared for using a national roundtable forum in the final year of the study, with subsequent email or virtual feedback from participants to summarise themes, recommendations and evidence-based practice gaps for sustainability and national scaling of Navicare [32]. ExpandNet resources will be used including the ExpandNet/WHO framework for scaling up [32, 44]. A scalability strategy will be co-developed with stakeholders using the ExpandNet/WHO Nine Steps for Developing a Scalability Strategy [32] if evidence collected from the four study sites supports success aligned with the study aims. The actions and findings throughout the preceding four EPIS phases will directly support this process and provide evidence of the scalability of Navicare nationally.

Data analysis

The co-primary implementation outcome of sustained access will be analysed quantitively using a regression model. A hybrid mixed methods approach integrating inductive and deductive qualitative analyses will be used for qualitative data. Inductive analyses will be informed by constructivist grounded theory [45] and deductive analyses will be conducted using framework analysis. These analyses will involve constant comparison (systematic comparison across participants) throughout the study and developing and charting data into a working framework [46]. The working framework will be based on the EPIS, study sites, and ExpandNet/WHO frameworks, the fidelity-adaptation tool [47], or other factors identified from the inductive analysis. Themes will be summarised to generate recommendations for scaling up Navicare or alternate mental health access and support programs, and to highlight evidence-based practice gaps.

Quantitative data analysed descriptively or using regression models will be combined with qualitative data to report on the implementation outcomes, context and contextual factors influencing implementation, and outcomes related to the process of applying EPIS (for example, variability in the timing of the exploration and preparation phases across sites prior to new site implementation) where relevant. As part of this mixed methods approach, qualitative and quantitative data will be used for sampling, triangulation, elaboration, expansion and development of the data [48]. The latter will involve using the findings to inform the development of implementation strategies. The process of combining the qualitative and quantitative data is expected to involve merging, connecting and embedding both types of data [49]. Trustworthiness of qualitative and mixed methods analyses will be maintained using field notes, reflexive journaling, memoing, and member checking of data [43, 45, 49]. Visualisation techniques (for example, causal loop diagrams and social network analysis) will be used to report on and provide feedback during the study on the study outcomes (for example, service access) and mechanisms of action of the implementation [50,51,52].

Cost of implementation will be evaluated across the study sites as well as for each EPIS phase and will include engagement of a local working group during preparation and implementation. Regression models will be used to explore the association between quantitative outcomes (for example, fidelity using the fidelity-adaptation tool with the implementation outcome of sustained access).

Implementation strategies

It is expected that multiple implementation strategies will be selected, implemented, and tailored to each site. These strategies are likely to include community and stakeholder engagement and access to local infrastructure. The Expert Recommendations for Implementing Change (ERIC) will be used to describe strategies, with FRAME-IS used to guide documentation on how strategies were adapted in the broader context of rural and remote Australia [31, 53]. Other potential strategies not captured by ERIC will be identified and described using the hybrid inductive and deductive data analysis approach (including the use of EPIS to identify barriers and facilitators).

Effectiveness study component methods

This study component will address aim 5.

Population

People with mental health related hospital data from public hospitals in the Greater Whitsunday Region and major regional hospitals situated just outside the Region will be included.

Data collection

Local and district hospital level data from the Queensland Hospital Admitted Patient Data Collection (QHAPDC) and Queensland Hospital Emergency Department Data Collection (QHEDC) will be obtained for new sites for the 24-months prior to commencement, during the implementation period of 12-months, and for 12-months in the post-implementation (sustainment) period. The longer length of pre-implementation data will be used to account for seasonal patterns in this control period. For the existing site, QHAPDC and QHEDC data will be collected for 12-months pre-implementation to avoid, as much as possible, the major disruptions due to the COVID-19 pandemic in the first half of 2020, and for the implementation period will be collected from October 2021 when the first client accessed the service. Post-implementation data will be collected from July 2025 at the existing site.

Co-primary and secondary outcomes

Outcomes will include time in an emergency department for mental ill-health (co-primary outcome), and mental health crisis-related Emergency Department attendances and admitted overnight hospitalisations for mental ill-health (secondary outcomes).

Data analysis

A pre-post comparison analysis will be used to examine if Navicare has a greater effect than any underlying secular trend. This will be achieved by fitting a generalised linear mixed model, to model the monthly rates of time in the emergency department (using a Gamma or normal distribution depending on the data), mental health emergency department presentations (using a Poisson distribution), and admitted overnight hospitalisations (using a Poisson distribution) [54]. Random intercepts will account for the non-independence of data from the same facilities. For sustained service access (co-primary outcome) and adoption, the trend before and after formal support for Navicare ends will be modelled (implementation to sustainment periods, Hypothesis 1). Residuals of the models used will be checked to assess the model’s validity, with histograms to check for bimodality and outliers, and with autocorrelation over time assessed with the Durbin Watson test, and adjustments for underlying trends and seasonality.

Economic evaluation methods

This study component will address aim 6.

Population

The economic evaluation will synthesise relevant service and population level resource use data.

Data collection

Program and associated implementation related resource use will be estimat