STRENGTHS AND LIMITATIONS OF THIS STUDY

This protocol provides a prospective view of the application of the Framework to Assess the Impact of Translational health research to the Centre for Research Excellence in Digestive Health (CRE-DH with the explicit aim of optimising research impact and providing direction for future digestive health planning and prioritisation.

This protocol describes three validated methods of impact assessment including the Payback Framework that describes impact using quantified metrics in different domains, economic analyses to quantify the return on research investment and narratives to describe the pathway to impact and provide qualitative evidence of impact.

There is always a lag in the health research translation process resulting in delays in reporting the full extent of research impact. This lag will limit the reporting of the longer-term benefits of the CRE-DH, for which evidence will not be available.

Introduction

Chronic gastrointestinal (GI) diseases are a major health burden in Australia and worldwide.1 2 More than one-third of Australians experience chronic or relapsing unexplained GI symptoms.3 4 In half of these cases, symptoms are serious enough to require a medical consultation usually at a general practitioner clinic or an emergency department. These cases also currently make up half of all referrals to GI specialists.5 For the majority of cases, however, no structural or biochemical abnormality is found after comprehensive and costly diagnostic workup resulting in a diagnosis of a disorder of gut-brain interaction (DGBI) most notably irritable bowel syndrome (IBS) or functional dyspepsia.6 7 Currently, there is no cure and for DGBIs treatment approaches are suboptimal, leading to frequent healthcare consultations by these patients.8 IBS alone has been estimated to cost more than US$41 billion annually in the USA.2 For other chronic GI conditions, including gastro-oesophageal reflux disease and inflammatory bowel disease (IBD), the prevalence is increasing, placing pressure on the healthcare system.9 10 Chronic GI diseases are also associated with significantly impaired quality of life, reduced work productivity, work absenteeism, relationship problems, higher levels of psychological distress and extraintestinal symptoms.11–16

While there have been impressive advancements into the underlying pathology of chronic GI diseases in recent years,17 18 there have been delays in the development of novel, pathology-based, subtyping of DGBI to facilitate improved integrated care and rationalised therapeutic strategies in clinical practice. This critical need was recognised by the Australian National Health and Medical Research Council (NHMRC) which funded the Centre for Research Excellence in Digestive Health (CRE-DH) from 2019 to 2024. The CRE’s vision is to advance the understanding, identification and treatment of chronic digestive diseases by implementing a risk-based and pathophysiology-based categorisation of patients and targeted treatments that are suitable for all sectors of the healthcare system (including primary care).

The specific objectives of the CRE scheme are to improve health-related outcomes and enhance translation of research outcomes into policy and/or practice while also building capacity in the health and medical research workforce.19 This is aligned with the NHMRC definition of the impact of research as ‘the verifiable outcomes that research makes to knowledge, health, the economy and/or society, and not the prospective or anticipated effects of the research’.20 However, the NHMRC also recognises that ‘the relationship between research and impact is often indirect, non-linear and not well understood and depends on complex interactions and collaboration across the health innovation system.20’ This emphasis on research impact arises from the growing pressure on grant funding bodies to be accountable for taxpayer-funded research and provide evidence of the wider benefits of research above and beyond traditional academic outputs (eg, publications). Examples include evidence of translation to new drugs and devices, changes to policy and practice and ultimately the social and economic impacts on society including the return on research investment, in order to support continued research funding.

In light of the complexities involved in assessing the impact from research, a myriad of Research Impact Assessment Frameworks (RIAFs) have been developed that provide a conceptual framework and methods against which the translation and impact of research can be assessed.21 22 However, most RIAFs tend to focus on specific research studies rather than research programmes such as CREs and are typically used retrospectively to justify past research investments. In contrast, the Framework to Assess the Impact from Translational health research (FAIT), developed by a team of health economists and health and medical researchers from the Hunter Medical Research Institute, is prospective in design and incorporates monitoring and feedback with the specific aim of increasing translation and impact.23 Ramanathan et al applied FAIT to the CRE in Stroke Rehabilitation and Brain Recovery and assessed its validity and feasibility.24 Overall, they found FAIT allowed a wide range of impacts to be reliably reported beyond the standard academic achievements. Thus, to take advantage of FAIT’s comprehensive design and prospective application, and allow for better benchmarking with other CREs, we have selected FAIT to assess the impact of the CRE-DH. This paper describes the protocol of a mixed methods study to:

Demonstrate the research impact and monetise the return on investment in the CRE-DH.

Provide a prospective view of optimising research impact.

Assess the suitability of FAIT.

The anticipated outcomes will be greater transparency and translation of research within CRE-DH, and the data will set the direction for future digestive health planning and prioritisation. In addition, this paper will contribute to this growing area of research impact assessment.

MethodsDesign

We prospectively applied FAIT to measure the impact of the CRE-DH. FAIT incorporates three validated methods of impact assessment. The Payback Framework describes impact within domains of benefit. Within FAIT, it has been modified to capture impact using quantitative indicators rather than qualitative data. Economic analyses are applied to quantify the return on research investment and narratives are used to describe the pathway to impact and provide qualitative evidence of impact. The assessment of the suitability of FAIT will take the form of a facilitated discussion among authors, at the conclusion of the impact evaluation, to identify the strengths and limitations of FAIT in the context of its application to the CRE and to make suggestions, if appropriate, for its future application

Details of FAIT have been previously published.23

Setting

The setting is the CRE-DH, which is composed of senior, mid-career, early career and student researchers, clinicians, consumers and other key stakeholders in the fields of gastroenterology, immunology, microbiology, epidemiology, dietetics, psychology and biostatistics primarily from four major research centres across Australia. These include the University of Newcastle and Macquarie University in New South Wales, Princess Alexandra Hospital and University of Queensland in Queensland, and Monash University in Victoria, along with substantial international contributions from the University of Leuven in Belgium, McMaster University in Canada, Mayo Clinic in USA and Kings College in the UK. The CRE-DH researchers pool their highly complementary expertise and capabilities for projects within the CRE-DH, which facilitates recruitment of large representative patient cohorts, the availability of cutting-edge methodologies and translation of findings into practice and policy. The CRE-DH was funded ($A2.5 million) from 2019 to 2024.

Participants

These include a mix of experienced, early career and student researchers associated with the CRE-DH and end users of the findings and outputs of the CRE-DH including other DGBI researchers, patients, consumers more broadly, clinicians, health services, policy-makers and industry partners.

Patient and public involvement

Development of the FAIT model involved extensive and broad end user engagement including interviews with the following key stakeholder groups—researchers from across the research spectrum, multiple Australian medical research institutes, health and medical research funders including the NHMRC, Australian Research Council, The Medical Research Futures Fund, NSW Office for Health and Medical Research, Brunel University, UK and Karolinska Institute, Sweden who were leaders in the field at the time and policy-makers. All interviews were conducted by staff from the Health Economics and Impact team at HMRI and covered attitudes to impact measurements, barriers and enablers, what was being done at the time and opinions about what should be done. There was a diversity of views and differences which were reconciled by designing a comprehensive framework (FAIT) that addressed all their needs. There is an absolute bias to selecting and reporting metrics for which there are data and this is addressed by impact planning that ensures as much data as possible is collected from the start. Other ways this bias is mitigated is by expressing the limitations and bias inherent in an impact assessment framework like FAIT.

This was supplemented by broad consumer representation on the CRE-DH advisory board that provided feedback at all stages of CRE-DH impact framework development. The use of the existing Payback domains and input from consumers with a range of conditions and experiences will ensure that the metrics selected reflect a broad range of potential impacts beyond academic impacts.

Procedure

The study involves a five-stage sequential mixed method design, summarised as follows:

Phase 1: Development of a programme logic model (PLM) to map the pathway to impact and establish domains of benefit and aspirational impacts.

Phase 2: Identifying and selecting appropriate, measurable and timely impact indicators for each of these domains and establishing a data plan to capture the necessary data.

Phase 3: Developing a model for the cost–consequence analysis and identification of relevant data for micro costing and valuation of consequences (where appropriate).

Phase 4: Determining selected case studies to include in the narrative including the data collection for these.

Phase 5: Collation, data analysis and completion of the reporting of impact using the three methods.

Phase 1: development of a logic model to map the pathway to impact and establish domains of benefit

A PLM is a critical component of any FAIT impact assessment. The PLM used in FAIT is a map that follows the pathway from the need for the CRE through its aims, activities, outputs and aspirational impacts. The CRE-DH logic model (figure 1) shows how the needs and aims drive CRE activities. These activities should produce outputs that, when used by an end user, creates an opportunity for the generation of impact. These impacts are articulated as both short-term and medium-long-term impacts under broad domains of benefit such as impacts on knowledge advancement, capacity building, clinical implementation, policy legislation, community and economic impacts. While the PLM appears linear, its application over the lifetime of the CRE-DH will most likely be non-linear and subject to change.

Figure 1

Logic model for the CRE-DH. CRE-DH, Centre for Research Excellence in Digestive Health; DGBI, disorder of gut brain interaction; GI, gastrointestinal; QOL, quality of life; TGA, Therapeutic Goods Administratio; EMCR, Early/Mid career researchers

Phase 2: identifying and selecting appropriate, measurable and timely impact indicators for each of these domains and establishing a data plan to capture the necessary data

The PLM (figure 1) identifies the Payback domains of benefits under which the CRE’s impact will be assessed. Impact metrics have been developed and customised for the CRE-DH taking into account their appropriateness for the CRE-DH and its aims and their ability to be measured in a timely manner. Table 1 shows the list of Payback metrics under each domain for which evidence is captured.

Table 1

Payback metrics table for the CRE-DH

Routine monitoring of implementation embedded into each project stream

The purpose of this data collection method is to collect quantitative data to monitor and measure the impact of specific studies within the CRE-DH and its capacity building and translational activities. Initial data collection involves annual distribution of a CRE-DH impact data survey via REDCap to chief investigators and associate investigators to be populated for all their CRE-DH affiliated researchers. Results of the survey are being collated into an Excel file that includes individual spreadsheets that are aligned with impact indicators. Additional data are being retrieved from available sources including publicly available online data from researchers’ university profiles, data collected for triannual CRE-DH advisory board meetings, through ethics systems, publication tracking and evaluation of CRE-DH organised capacity building and translational activities. The Excel spreadsheets for each project stream are being emailed annually to each CI to add any data that has not been captured using the above methods.

Reports during the regular team meetings

This data collection method aims to collect quantitative and qualitative data to monitor and measure the translation, implementation and impact of CRE-DH that are not obtained from routine monitoring. The data are collected online by accessing the recorded monthly CRE-DH meeting minutes and added to project stream spreadsheets or flagged for further discussion in semistructured interviews for vignettes or case study examples of CRE-DH impact, described as part of phase 4.

Phase 3: developing a model for the cost–consequence analysis and identification of relevant data for microcosting and valuation of consequences (where appropriate)

To determine whether the cost associated with the delivery and participation in activities associated with the CRE-DH and the consequences achieved represent a good return on investment, a cost–consequence analysis will be undertaken.25

First, we will detail out the activities funded by the NHMRC investment. Second, we will microcost any activity and other costs not covered by the US$2.5 million NHMRC research investment and add these to the NHMRC investment as implementation costs. This will include costing all in-kind investigator time and capacity building participation time not directly funded by the CRE monies.

Microcosting data will involve a log of all intervention activities including the individual’s involved, their roles and wages and the time taken for implementation. Other resources such as travel and consumables will also be costed. The proportion of cost attributable to CRE-DH activity will be estimated where feasible.

In collaboration with the lead investigators of the CRE-DH, the consequences of the CRE-DH will be established including the consequences that cannot be monetised and appear in their natural units in the Payback metrics table. For those consequences that can be monetised, economic methods will be employed to adequately monetise their value and determine the appropriate level of attribution to the CRE-DH. This will include a search of the literature for established values for these consequences (where they occur), clearly defined assumptions about these values and sensitivity analyses to account for any variance in these values. Given that CRE-DH activity will be occurring concurrently with other research activities supported by the research institutions from which CRE-DH researchers are affiliated, attribution of consequences (eg, leveraged funding) will take this into account. Where practical, researchers will be asked for their own assessment of CRE-DH attribution to a particular consequence or a conservative attribution percentage will be applied to avoid overclaiming the consequences and impacts of CRE-DH. All values will be converted into Australian dollars and valued in the year that the final analysis is conducted.

Phase 4: determining selected case studies to include in the narrative including the data collection for these

During the course of the CRE-DH, the pathways to adoption of the outputs will be documented by the team and team meetings will be used to highlight potential case studies that can be developed to demonstrate outstanding impacts of the CRE-DH or case studies that describe key learnings. Semistructured interviews will be conducted to collect relevant data that will inform these case studies. It is anticipated that these interviews will be with CRE-DH researchers and key end users, where appropriate.

Semistructured interviews involving CRE-DH staff, collaborative investigators, advisory group members and other key stakeholders

Qualitative data will be collected, to provide context and a richer, more comprehensive overall understanding of the impact of the CRE-DH. Topics of interest will be flagged through the quantitative data collection and in meeting discussions, based on the underlying question of ‘How did this publication, conference presentation, collaboration, capacity building activity or project lead to an impactful outcome that would not have been achieved without the CRE-DH?’ Interviews will be facilitated by the HMRI FAIT team, who have expertise in qualitative data collection for impact evaluation. These data will be narratively synthesised and triangulated with quantitative data and incorporated into impact evaluation reporting within the narrative method and include specific quotes from the researchers and end-users.

Timing

Impact assessment data will be collected for the 5-year period from November 2019 to October 2024.

Phase 5: collation, data analysis and completion of the reporting of impact using the three FAIT methods

The data collected over the course of the CRE-DH using the various methods described above will be reported using the FAIT scorecard format.23

Results for the metrics table will be collated and where bibliometric results are required, a cut-off date will be established after which time, the results will not be updated. The cost–consequence will be reported by way of a cost–consequence table that will only include the consequences that can be monetised. Other consequences will be reported in their natural units in the Payback metrics tables. The narratives will be reported as vignettes highlighting some of the outstanding achievements of the CRE-DH including the pathway to translation and impact.

Ethics and dissemination

This impact evaluation study has been registered with Hunter New England Human Research Ethics Committee as project 2024/PID00336 and ethics application 2024/ETH00290. Results of this study will be disseminated via medical conferences, peer-reviewed publications, policy submissions, direct communication with relevant stakeholders, media and social media channels such as X (formerly Twitter).

Discussion

This protocol aims to define and describe processes to collect, collate and synthesise data for the CRE-DH to evaluate the impact of the CRE-DH from inception in November 2019 to final data collection in mid-2024 for reporting of outcomes in October 2024. We plan to operationalise this protocol as a mixed-methods study by applying a PLM to the original aims and needs identified in our CRE-DH application, to use that modelling to review CRE-DH progress towards our aims, and to inform prospective direction for the CRE-DH based on ongoing progress and at specified annual data collection review time points. Therefore, our impact evaluation will be an organic, prospective, informative and responsive process, as well as providing an overall final and retrospective account of CRE-DH impact by the end of 2024. Impact will be reported and used to inform future funding applications and direction for digestive health research in Australia, and position the CI, AI and affiliate team as leaders in the field internationally. This impact evaluation will also inform future directions for DGBI and other digestive diseases research, which we expect to overlap and integrate more with related fields such as immune and microbiome research in coming years. The prospective design of our impact evaluation will facilitate expansion into new fields throughout the life of the CRE-DH, which will enhance translation potential, impact and transformative research and clinical practice change.

Although, there are other frameworks from various medical fields26 to assess evaluation of research outcomes, this evaluation applied the FAIT to the CRE-DH with the explicit aim of optimising research impact and providing direction for future digestive health planning and prioritisation.

Despite the benefits of comprehensively assessing the impact of the CRE-DH using three distinct methods namely quantified impact metrics, a cost–consequence analysis and a narrative of the impact there are some potential risks and limitations. These include (1) Lag in translation could impact on the ability to capture and demonstrate longer-term impacts. (2) Data collection for impact reporting while feasible, does require additional commitment by CRE partners to ensure it is comprehensive and complete. Therefore, this could be seen as an added administrative burden and may not be completed as required. However, the desire to continue the collaboration and the fact that CRE affiliates have been engaged with the impact assessment from the start should provide a counterbalance to the burden. The inclusion of the HMRI Research Impact Team as expert advisors will also ensure that multiple strategies previously used in other CRE impact assessments are employed to enhance data collection. (3) Attribution of impacts is challenging and will have to rely on researchers to attribute the contribution of CRE-DH to a particular consequence. (4) Selection of case studies means other potential impact stories may be foregone.

The novelty of this work is that the application of FAIT is still very much in its infancy with only two protocol papers (both using very different framings for the application)24 27 and only one results paper published.28 There is still much to learn and reflect on in the application of such a comprehensive framework, and this protocol paper will provide a useful roadmap for other GI research collaborations planning formal impact evaluations. A deepened understanding about what enhances the impact of a CRE will only be possible when we have benchmarked protocols and outcomes. We will then have the ability to undertake meta-analyses to ascertain what works under what circumstances in order to further enhance the impact in a large and complex research collaborative such as a CRE. Contribution to a larger bank of metrics will give visibility to the potential capacity and capability impacts from CREs.

Conclusion

This study will capture outputs and impacts that have been initiated or enhanced as a result of the CRE-DH’s collaborative efforts of basic scientists, allied health and medical clinician researchers, translational scientists, consumers and advisors across the spectrum from animal, preclinical laboratory research to health service delivery from acute to integrated and primary care settings. All costs for CRE-DH activity will be valued and where possible, the economic analysis will monetise reportable CRE-DH outcomes and impacts. If this is not possible, these impacts will be reported in their natural units. We expect this impact evaluation to comprehensively describe the contribution of the CRE-DH to a range of impacts including any improved outcomes for people suffering with chronic and debilitating digestive disorders. The impact evaluation will inform future directions for digestive health research and assessment of its impact.

Ethics statementsPatient consent for publication

Not applicable.