The KOL interviews consisted of a total of 27 interviewees from which, 11 (40.7%) individuals were from regulatory bodies, 11 (40.7%) were from industry, 4 (14.8%) from academia and 1 (3.7%) from a patient advocacy group. The interviewees largely comprised clinicians, scientists and statisticians, see Table S1 (electronic supplementary material) for more details.

Each interviewee was administered 5 questions (Q1–Q5). A full description of the 5 questions can be found in the electronic supplementary material (Section 1). Key findings from the qualitative assessment of the responses for the 5 questions, including the themes and sub-themes identified for each specific question, are summarized below and presented in Table 2.

For each question, the most prevalent themes based on the quantitative analysis are presented in Figs. 1, 2, 3, 4 and 5 and discussed below along with key findings from the qualitative content analysis.

In question 1 (Q1), interviewees were asked about the future application of BRA. They were asked about what should be achieved in BRA, and how that should be achieved over the next 5 years.

The bar graph (Fig. 1) uses a color-coded legend to illustrate different themes and corresponding color bars to represent sub-themes under each of these themes. For example, the blue color represents the theme Current status of BRA and the blue bars represent the corresponding sub-themes, Minimal to no usage currently, Emerging only, Some measures in place but more required and Already well embedded with dedicated groups, etc. A similar legend was used to identify themes/sub-themes for the remaining questions Q2–Q5.

Bar graph summarizing the percentage of interviewees who contributed to a given sub-theme for Question 1. We know there is growing interest in not only what should be achieved in benefit–risk assessment (see M4E R2 and E2C R2), but also in how we achieve that. How do you think this will proceed over the next 5 years for your organization and for development of medical products within the wider medical product sector? BR benefit–risk, BRA benefit–risk assessment

Regarding the theme of Current status BRA, the most frequent sub-theme Some measures in place but more required was highlighted in the comments provided by 70% of the interviewees, while Minimal to no usage currently was highlighted in the comments provided by only 4% of the interviewees. For the theme Step to the future—value demonstration, the most frequent sub-theme was Regulatory authority expectations, potential for further guidance based on comments provided by 78% of the interviewees suggesting the need for regulators to set expectations and provide further guidance. For the theme Step to the future—development of process (e.g., working practices and procedures), the sub-theme Process guidance by expert groups was the most frequent and was highlighted in the comments provided by 63% of the interviewees, while comments by 48% of the interviewees contributed to the sub-theme Incorporation of BRA into other (e.g., existing) processes. Here interviewees highlighted the value of dedicated BRA expert groups in guiding best practice within industry, and the need to have a clear, structured, and well-integrated BRA process, with BRA helping to inform decision-making throughout the product lifecycle. For the theme Step to the future—development of expertise, the sub-theme Establishment of BRA methodologies and refinements was highlighted in the comments provided by 67% of the interviewees, which was the only identified sub-theme within this theme. Interviewees suggested a need to improve methodologies and tools over the coming years with qualitative assessment of BR as the foundation supplemented by quantitative methods as appropriate (Fig. 1).

In Q2, interviewees were asked about the value of planning in the execution of BRA and to indicate how and when planning should occur in the medical product life cycle based on their own experience.

Regarding the theme Value of planning, the sub-theme Identification of best approach/fit for purpose was the most frequent and was highlighted in the comments provided by 56% of the interviewees. Interviewees indicated the importance of identifying the best approach for BRA along with early planning to ensure appropriate trial design to facilitate the BRA (for the theme Level of planning activities, comments provided by 63% of the interviewees contributed to the sub-theme Trial design level). The need for interactions between pharmaceutical and medical device companies and regulators starting during early development and continuing throughout the product lifecycle was also highlighted (for the theme When to start planning comments provided by 44% of the interviewees contributed to the sub-theme Phase 2, and the Importance of BRA throughout the product lifecycle was highlighted in the comments provided by 33% of the interviewees). For the theme Plan review and update, the sub-themes Milestone driven and Ad hoc based on significant new data becoming available were the most frequent sub-themes and were both highlighted in comments provided by 44% of the interviewees, suggesting the BR planning needs to be milestone-driven and updated “ad hoc” as new data becomes available (Fig. 2).

Bar graph summarizing the percentage of interviewees who contributed to a given sub-theme for Question 2. Planning and execution are two key steps of any clinical development endeavor. How do you see the value of planning in the benefit–risk assessment activities and what have you learnt from your own experience? Please articulate whether, for example, you see this planning as solely a compound/indication level activity or also a trial design activity. When during the lifecycle should planning occur? How should plans be maintained and updated over time? BRA benefit–risk assessment

In Q3, interviewees were asked about the integration and alignment of safety and BRA. They were also asked about the criteria used to determine when plans should be made for additional assessment of benefit or risk, including the information sources that should be considered and the role of observational data.

Regarding the theme of Integration of safety and BRA, the most frequent sub-theme was Alignment between processes (e.g., working practices and procedures), which was highlighted in the comments provided by 59% of the interviewees. Interviewees indicated that safety should not be viewed in isolation and there is a need for more interaction between the benefit and risk functions within companies. They considered that BR should be viewed as the overarching concept with relevant processes aligned under this umbrella. Where feasible, the processes for BRA and Health Technology Assessment (HTA) should be aligned. For the Criteria for additional assessment theme, Upon identification of significant safety concerns and Based on availability of new data and its relative value were the most frequently reported sub-themes and were highlighted in the comments provided by 41% and 37% of the interviewees, respectively. Post-marketing, observational and clinical data sources were the most frequently identified sub-themes under the theme Sources of data for BRA, with Observational data considered Of some value based on the comments provided by 48% of the interviewees and Very valuable by 30% of the interviewees. Observational studies were thought to be of particular value in the post-marketing setting. Interviewees mentioned the frequent use of observational studies to address safety questions and also identified an opportunity to utilize observational data to evaluate benefit/effectiveness post-approval. Potential drawbacks of observational data such as bias, database limitations and a need to develop the right analytical tools were stated (Fig. 3).

Bar graph summarizing the percentage of interviewees who contributed to a given sub-theme for Question 3. Risk and Risk Management are key components of benefit–risk assessment. How do you see safety assessment and benefit–risk assessment being better integrated or aligned? All products require ongoing monitoring for safety, even if just through passive event reporting. What are the criteria used to determine when plans should be made for additional assessment of benefit, or risk (beyond routine pharmacovigilance), to demonstrate that the product has, or continues to have, a favorable benefit–risk? What are the sources of information that should be considered? What role does observational data have? RWD real-world data

In Q4, interviewees were asked about planning of activities for assessing, expressing and communicating BRA activities throughout the medical product lifecycle. Interviewees were also asked what kind of activities these could be. Finally, interviewees were asked to comment on the important priorities concerning BRA in the future.

With regard to the Planning and assessing of BR activities theme, the sub-theme of Clarity on the task and objective of the BRA was the most frequently reported sub-theme and was highlighted in the comments by 44% of the interviewees. Interviewees communicated the importance of investing time upfront to fully understand and define the aims and objectives of the BRA before proceeding further.

For the Priorities regarding BR (in the future) theme, Understanding BR value at different stages of the lifecycle was the most frequently reported sub-theme and was highlighted in the comments provided by 44% of the interviewees. With respect to this sub-theme, interviewees discussed the importance of starting BRA early in development to ensure appropriate selection of the target population/indication, characterization of risks, trial design, and to address gaps and minimize uncertainty. It is too late to act if gaps are only identified at the time of submission. Interviewees stated that the type of analysis performed may differ depending on the lifecycle stage, amount of data available, therapeutic area (e.g., BR thinking may be different for severe conditions with a high unmet medical need, such as oncology indications, compared to other therapeutic areas) and whether the BR profile is straightforward or more challenging. A relatively rudimentary assessment may be performed at the early stages of the product lifecycle, which then evolves over time as more information becomes available. Having a dedicated safety management/BR team to support the BRA through the product lifecycle was considered helpful.

With respect to the theme of Expressing and communicating BR, Understanding the audience (e.g., clinicians, patients, or regulators) and Appropriate contextualization of benefits and risks were the most frequently reported sub-themes and were highlighted in the comments of 44% and 41% of the interviewees, respectively. Interviewees indicated a need for better communication of BR information in a way that can be easily understood by all stakeholders including patients and considered that consultation with communication experts and more education/training on effective methods of communication may be helpful in this regard. Experience gained with assessing and communicating the benefit-risk of COVID-19 vaccines could also be leveraged. Interviewees recommended using the latest technologies and platforms such as smart phones and social media as a way to disseminate information. It was also considered important to share information on BRA methodologies and decision-making tools through publications. In terms of appropriate contextualization of benefits and risks, interviewees mentioned the use of tools such as value trees [25], the appropriate structuring of document templates, and adoption of an integrated approach across different company functions in order to develop a unified perspective on BR rather than the benefits and risks being evaluated and presented separately. The potential to compare BR profiles across products was also mentioned along with the possibility of developing measures/metrics that are more readily understood by clinicians and patients (Fig. 4).

Bar graph summarizing the percentage of interviewees who contributed to a given sub-theme for Question 4. Using the provided Figure (see Fig. 1 in Appendix) as a reference model for reflection, how will you be planning activities to assess, express, or communicate benefit/risk at different times of the product lifecycle such as when: developing a target product profile?, seeking a clinical trial application?, discussing pricing and reimbursement with payers?, preparing communication materials for health care professionals and patients?, submitting a dossier for a new indication for an existing approved product? What kind of activities could these be? How do you see these activities changing? What do you think will be important priorities concerning benefit/risk assessment in the future? BR benefit–risk, BRA benefit–risk assessment, HTA health technology assessment, RWD real-world data

In Q5, interviewees were asked about patient engagement in BRA planning. Regarding the theme of Value of patient input, the most frequent sub-theme was Utilize patient experience/preference to guide new product development and BRA, which was highlighted in the comments by almost all (96%) of the interviewees. This was followed by the sub-theme of Understand patients’ tolerance for risk and trade-offs, which was highlighted in the comments provided by 74% of the interviewees. Interviewees indicated that patients should have a voice and be involved early in the product development process. They can provide valuable input into various aspects of development such as drug candidate selection, clinical trial design, benefit-risk assessment, and product labelling. Factors important to patients such as convenience of the dosing regimen, should be identified. Interviewees indicated that the focus on patient engagement will continue to grow in coming years and developing a common approach across industry would be beneficial. Companies may work together to understand what is important to patients within a given therapeutic area.

Regarding the theme of Data collection/input, How to gather information (from patients) was highlighted in the comments provided by 78% of the interviewees and was the most frequent sub-theme. The sub-themes Selection and development of appropriate methodologies (to solicit patient input) and When to start involving patients were highlighted in the comments provided by 59% and 41% of the interviewees, respectively.

Interviewees indicated that patient input can be sought via patient organizations and advocacy groups and patient representatives can be included on advisory and steering committees. The importance of learning about individual patient perspectives was highlighted. Potential methods to solicit patient input such as qualitative interviews, surveys, diaries, patient journey mapping activities, direct ethnographic observation, risk perception studies, BR preference studies/elicitation methods, Patient Reported Outcomes (PROs), Patient Preference Information (PPI), and Clinical Outcome Assessment (COA) were described. Interviewees highlighted the need to carefully consider the methodology and analysis to ensure the data collected from patients add value rather than creating noise. The time required to develop innovative endpoints/methods and seek regulatory agency buy-in should also be considered. With respect to communicating BR information, interviewees considered that visual presentations and interactive tools that are tangible to patients should be used (Fig. 5).

Bar graph summarizing the percentage of interviewees who contributed to a given sub-theme for Question 5. How can patients be involved in benefit-risk assessment planning? How should the collection of patient experience and patient preference data be incorporated within benefit–risk planning? BRA benefit–risk assessment, PPI patient preference information, PROs patient reported outcomes