3.1 Literature Search and Study Selection

The study selection is illustrated by a Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) flow diagram in Fig. 1. The number of identified records was 282. They were retrieved from the following databases: EMBASE (n = 211), MEDLINE (n = 67) and Cochrane Library (n = 4). A total of 64 duplicate records were removed. Moreover, 172 records were excluded during screening of the abstract and title, which left 46 studies for full screening. Of those, 11 were excluded because of not containing components for economic evaluations (n = 5) or systematic reviews (n = 4), or because they were abstracts (n = 2). Overall, 35 reviews from the database searches were deemed eligible for inclusion as listed in the ESM under literature search results.

Fig. 1

Preferred reporting items for systematic reviews and meta-analyses (PRISMA) flow diagram

The hand search yielded five papers, of which four were excluded for the following reasons: no component of economic evaluations (n = 1) or systematic reviews (n = 2), and for not being concerned with orphan drugs (n = 1). It meant that one paper was carried forward from the hand search, which brought the total number of eligible reviews to 36. The ESM lists papers included for full screening.

3.2 Study Characteristics

A two-step approach was used to determine if studies could qualify as systematic reviews. First, a Scottish Intercollegiate Guidelines Network search filter for systematic reviews was used, which is a pre-tested search strategy that identifies the higher quality evidence from vast amounts of literature indexed in a medical database. Second, eligibility was assessed, and a consensus obtained between the first and second reviewer on their inclusion. Using this approach, two scoping reviews were included because the methods were sufficiently systematic [31, 32]. Similarly, a study described their approach as a series of targeted literature reviews, which was also sufficiently systematic for inclusion [12]. The number of records included in the systematic reviews varied between 2 and 338. The ESM provides a summary of study characteristics.

3.3 Critical Appraisal

One study had low quality [31]. The highest frequency was found in the category of moderate quality, which comprised 27 studies [12, 32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57], whereas eight studies were rated as having high quality [58,59,60,61,62,63,64,65]. The ESM includes scores for each individual Joanna Briggs Institute checklist question across all studies, which showed that most studies (n = 35) obtained points from question 4, which was: Were the sources and resources used to search for studies adequate? Question 8 was not widely applicable and was fulfilled by the least studies (n = 4). Question 8 was: Were the methods used to combine studies appropriate? Critical appraisal methods used in individual systematic reviews were assessed by question 5: Were the criteria for appraising studies appropriate? Fourteen studies included appropriate criteria for critical appraisal, whereas in 13 studies it was unclear whether they did, seven studies did not, and for two studies the question was not applicable.

3.4 Data Extraction

The systematic reviews were divided into two categories: those that considered a specific rare disease (13 studies) and those that considered multiple rare diseases (23 studies). As shown in Fig. 2, three broad themes were identified: issues with health economic parameters, issues with health economic evaluations and issues with estimating value/reimbursement, with subtopics further developed for each theme. For issues with health economic parameters, the subtopics were the natural history of disease, clinical effectiveness, costs and quality of life. For issues with health economic evaluations, the subtopics were cost effectiveness and budget impact. For issues with estimating value or reimbursement, the subtopics were thresholds, value frameworks and multiple criteria decision analyses (MCDA). A repository of all extracted data on issues for economic evaluations of rare diseases is available in the ESM.

Fig. 2

Data extraction themes, sub-topics and findings

3.5 Issues with Health Economic Parameters3.5.1 Natural History of Disease

Rare diseases often progress slowly or are chronic by nature, which make clinical trials insufficient as they tend to have short durations [12, 62]. The non-existence or limited number of studies that include data on prevalence and incidence further magnify issues [44, 48, 51]. Moreover, clinical experts are few and private practitioners may only encounter few rare disease cases, which make them difficult to diagnose, and expert advice on rare diseases might not be easy to find [12, 44, 45, 57]. Delayed diagnosis and misdiagnosis make it difficult to define treatment-eligible cohorts [45, 50, 51, 58].

To summarise, an economic evaluation is challenging, for example for long-term modelling, because of missing data on the natural history of the disease or unknown rare disease trajectories [40]. Although registries can alleviate data issues, they may suffer from challenges such as diverging disease and diagnostic codes, data ownership and missing comparator data [12, 48, 50].

3.5.2 Clinical Effectiveness

Whilst clinical trials are common sources for effectiveness data in economic evaluations, appropriate clinical evidence is not always available for this purpose [57, 58]. Moreover, clinical trials may suffer from short durations, small sample sizes, premature termination, inadequate power, missing data or missing control arms, for example, for ethical reasons [12, 37, 45, 47, 57]. In addition, published long-term studies providing post-marketing data on safety and efficacy are rarely available [37, 38].

Other challenges are missing treatment guidelines, data to predict treatment responses, concerns on the patient relevance and the use of surrogate endpoints [40, 50, 52, 60, 63]. Comparator data are essential for economic evaluations, but might be missing for rare diseases, and if they are available, there might not be a consensus on the use of treatment regimes or treatment eligibility of patients, which result in heterogeneity across studies [39, 48, 50, 51]. A review found that studies reporting clinical evidence for orphan drugs had low-to-moderate quality, and none of them had high quality [60].

3.5.3 Costs

Cost-of-illness or burden-of-disease studies are scarce in rare diseases [12, 34, 39, 42, 43, 48, 52, 55]. Of those studies available, most are retrospective and only a small proportion of studies report indirect, non-medical or informal-care costs. [12, 34, 51, 58, 59]. Aggregated primary data are rarely available, hence, studies tend to report patient-reported claims or registry data [42].

It is complicated to transfer cost-of-illness results between different rare disease settings because of differences in study designs, methods and results. For example, one study estimated lost productivity without following recommendations for handling uncertainty [42]. A multitude of factors influence transferability such as data sources, geographical perspective, nomenclature, assumptions, discount rates, unit costs, treatment guidelines and value frameworks [34, 43, 46, 50].

3.5.4 Quality of Life

Quality-of-life studies in rare diseases are limited, but availability depends on the rare disease of interest [35, 39, 47]. For example, a review found two studies that included utility values for Cushing’s syndrome, whereas another review concerned with Crigler–Najjar syndrome found no data on the humanistic burden, apart from anecdotes on treatment challenges [39, 47]. In addition, there are data limitations on the quality of life of caregivers [63]. A probable explanation for the scarcity is the limited applicability of quantitative methods such as choice experiments or conjoint analysis in rare diseases, for example because of small sample sizes [35]. Furthermore, studies tend to be small, not randomised or controlled, which decreases the reliability of conclusions [