The results provide an overview of the tool (available online at [30, 31]), the analysis of the Swedish HTA system and processes, and outcomes of the expert workshop.

The final version of the evidence generation tool consisted of a flowchart with two main processes, evidence evaluation and generation (Fig. 2) where each step links to additional information. The evidence evaluation process (Fig. 2 – left side) assesses for the highest feasible level of study design and consistency across studies. If issues related to the existing evidence are identified, the user is directed to the evidence generation process.

The overview flowchart of the evidence generation tool. Users start from the yellow diamond and work their way through the two processes on evidence evaluation and generation. The i-symbols indicate links to further information and resources. A high-resolution figure is available in the Supplementary Material (‘Fig. 2 – high resolution’)

The evidence generation process (Fig. 2 – right side) guides the user through study planning, highlighting major challenges. For instance, consistent patient and control groups across studies to ensure inclusion of data in HTA, early involvement of patients/users in design to inform effect measurements (e.g., patient-reported outcomes, quality of life, organisational and patient ethical aspects), identifying clinical research partners that meet requirements, and validating measurement techniques. The tool guides the user on selection of measurements of outcome and other indicators of interest related to organisation, patient ethics and health economics.

Figure 3 provides an overview of the developed website. This includes the homepage with a tutorial video, project information, and a short user guide. Users can navigate to a downloadable, interactive pdf-version of the evidence generation tool; a searchable database of the HTA analysis; the Swedish HTA system actor map; and, the HTA aspect maps (clinical measurements of outcome, health economic, organisational, and patient ethical aspects).

Diagram of Evidence4Med.Tech, the dedicated website for the developed evidence generation tool and its accompanying material

The pdf-file of the tool includes the process map with accompanying information. The searchable database contains summary information based on the analysis of the 65 regional HTA reports, allowing searching for products, clinical research questions, patient populations, reasons for excluding studies, and more. The Swedish HTA system map gives an overview of the main actors, their relationships and links to external information. The HTA aspect maps provide examples of clinical measurements of outcome, health economical, organisational and patient ethical aspects, considered in the analysed regional HTA reports.

Seven experts participated in the user evaluation of the tool, a summary is given in Fig. 4. Experts had a background in the medical device industry (N = 3 experts), academic research (N = 3), healthcare (N = 2), and public organisations (N = 1). All participants had experience from various stages of medical device development, including research and development, clinical investigation, regulatory requirements, quality assurance and marketing.

Summary of the user evaluation of the evidence generation tool as assessed by seven users according to the Technology Acceptance model. The diagram shows the overall user response (categorised as positive, neutral or negative) and a summary of the feedback

Five experts responded positively on the perceived usefulness of the tool and two neutrally. The experts stated that the tool was useful for creating understanding of evidence generation and evaluation, communicating project status, and carrying out new studies. However, it was raised that the tool required sufficient background knowledge to be useful.

On perceived ease of use, five experts responded positively, one neutrally and one negatively. Positive aspects included that the tool provides an overview of important steps, linking to useful resources. Criticism related to the knowledge requirements on evidence generation and HTA, and the visual presentation.

All experts responded that the tool was a good idea. According to the experts, the tool presents information on several topics (e.g., evidence generation, evidence and study design aspects) where knowledge is limited, particularly in start-up companies and the engineering profession. Following the tool would increase the probability of generating sufficient evidence. Concerns included the required background knowledge of the user, how to ensure that information stays updated, and the need for a trusted organisation to take ownership to reach a wider audience.

Regarding behavioural intention, four experts were positive, three were neutral. Expressed reasons for use were to navigate the landscape without reinventing the wheel, improve efficiency and planning, communicating with colleagues, and educate on HTA. One of the concerns raised by an expert was what to do in the case of a negative decision based on a HTA.

Figure 5 shows the principal actors in the Swedish HTA system. Central are SBU [32], the HTA network [33], regional HTA organisations and healthcare regions [33], MTP Council [34], TLV (responsible for decisions on subsidy for pharmaceutics, dental care procedures and medical devices, and health economic assessments) [35], LV (the Swedish competent authority for medical devices and medicinal products) [36], the Swedish Association of Local Authorities and Regions (SKR; a politically run organisation representing the local governments of Sweden) [37], the National System for Knowledge-Driven Management and the National Programme Areas (NPO’s) [38], the New Therapies (NT) Council [39], and the National Unit for Odontology (HTA-O) [40].

Map of the key actors in the health technology assessment system, Sweden. Elements indicate actor and connections indicate relationship. The elements are colour coded by category of actor. HTA: health technology assessment; HTA-O; LPO: local programme area; LV: Medical Products Agency; MedTech: medical technology; MTP: medical technology products; NPO: national programme area; RPO: regional programme area; SBU: Swedish Agency for Health Technology Assessment and Assessment of Social Services; SKR: Swedish Association of Local Authorities and Regions; TLV: The Dental and Pharmaceutical Benefits Agency. A high-resolution figure is available in the Supplementary Material (‘Fig. 5 – high resolution’). An interactive map is available in Kumu at: https://embed.kumu.io/4a71e0b9942fa9c8bec97c6bd5970817

SBU [32] is the government agency responsible for carrying out independent assessments of methods and interventions in healthcare, dental and social care. The agency acts head and secretary for the HTA network and develops the handbook on systematic reviews and assessments. The HTA network facilitates collaboration between the organisations, harmonises and develops HTA methods [33]. The network includes representatives from the regional HTA organisations, TLV, the National Board of Health and Welfare and the Public Health Agency of Sweden. The MTP Council [34] acts as expert reviewer and is mandated to give recommendations on the use of certain medical technologies. The council decides on national introduction of medical technologies, assessment requests from member organisations, recommendations on products introduced on a national level, and handles communication of follow-up study outcomes, and more.

The NPO’s [38] lead knowledge transfer in their respective medical fields and have regional (RPO’s) and local level (LPO’s) counterparts across Sweden.

Regional HTA organisations engage with patient representatives on a case-by-case basis. In communication with regional HTA centres, it was stated that patient involvement was limited. This is further supported by the literature. Werksö and Andersson reported that Swedish HTA organisations underutilised patients and their perspectives. Where, SBU and TLV involve patients at different stages of HTA (e.g., reviewing reports, setting priorities and decision-making). The MTP council has a member advising on patient involvement [41].

Figure 6 summarises the main HTA processes in Sweden conducted by SBU, TLV and regional HTA centres. During 2016–2020, 65 regional HTA reports were published related to medical devices on request of regional healthcare organisations or the MTP council. During the same period, TLV published six health economic evaluations on medical devices for the healthcare regions, NT Council and Swedish Board of Health and Welfare. MedTech companies can initiate HTA through TLV and submit complementary evidence and health economic analysis for consideration. Three out of six reports included health economic analysis by the manufacturer. SBU published three evaluation reports, two preparatory reports, and 19 information service reports (summaries of peer-reviewed publications). The MTP Council published six recommendations related to medical devices.

Main processes for health technology assessment (HTA) and organisations involved in HTA in the Swedish system. A high-resolution figure is available in the Supplementary Material (‘Fig. 3 – high resolution’)

Figure 7A shows the most common reasons for excluding studies during regional HTA. This included instances where the PICO (patient group, intervention, control, outcome measures) did not correspond to the definition in the assessment. Other reasons for exclusion included: wrong study design (e.g., a study did not meet the minimum criteria for study design, mainly case series), wrong publication type (e.g., non-peer reviewed conference abstracts), low study quality or few study participants. The category’Other’ mainly included studies where the data of interest could not be extracted from the publication for the purpose of meta-analysis, for example when data for a specific subpopulation could not be separated from the overall dataset.

Aggregate analysis of regional HTA reports. A: reasons for excluding study data. B: studies defined in the inclusion criteria for published evidence. Based on analysis of 65 regional HTA reports published between January 2016 and end of December 2020. HTA: health technology assessment; PICO: patient group (P), intervention (I), control (C) and outcome(s) (O); RCTs: randomised controlled trials

Evidence grading is based on GRADE (Grading of Recommendations Assessment, Development and Evaluation) [22, 42], impacting study inclusion and exclusion criteria during regional HTA. Systematic reviews, randomised controlled trials and cohort studies were regularly included, excluding case studies and series (Fig. 7B). Healthcare registry data was included in isolated cases. The minimum number of study participants varied between five and 500 participants, or a median of 20 participants per study or study arm.

Figure 8 shows the coded main aspects considered during HTA according to the analysis. The main dimensions were clinical effect/main research question, organisation, health economics and patient ethics. Organisational queries could be divided into five categories: the current state of the organisation and technology use, resources and the impact of the technology on available resources (e.g., staffing, use of facilities, supporting services, and training), routines and implementation factors. Health-economic queries were categorised into health economic evidence, cost-effectiveness, and economic impact on the organisation. Patient ethics concerned current evidence, patient factors (e.g., quality of life and integrity) and organisational impact (e.g., displacement effects and ethics of priority).

Aspects related to organisation, health economics and patient ethics, considered during health technology assessment (HTA) based on the analysis of published regional HTA reports. A high-resolution figure is available in the Supplementary Material (‘Fig. 8 – high resolution’). An interactive presentation is available in Kumu at: https://adamdarwich.kumu.io/aspects-considered-during-health-technology-assessment-hta-in-sweden

Table 2 summarises identified issues and proposed solutions on HTA and evidence generation for medical devices based on the expert workshop. In Sweden, differing evidence requirements during healthcare procurement and HTA result in medical devices being procured and later subject to a negative recommendation from the MTP Council. This highlights the need for better coordination between HTA and procurement, and improved follow-up in healthcare. There is a need for tailoring evidence grading to medical devices as these tend to exhibit shorter product iteration cycles and smaller target populations as compared to medicinal products. Other study design challenges include blinding and accounting for learning curves [43, 44].

Experts expressed the value of adopting an individualised approach to medical devices. This could improve study design and evidence generation, with increased focus on the patient/end-user experience. This requires new financing initiatives on implementation research.