To meet the aim of our study, we designed a discrete choice experiment (DCE) that was administered online by Motivaction, a professional sampling agency located in the Netherlands, in June 2022. This agency recruited respondents from their online panel and quota sampled them to be representative of the public in terms of age (18–80 years), sex, and education level.

Respondents received a participation fee of 2.30 euros upon completion of the questionnaire, which they could save in a personal account or donate to charity. The Research Ethics Review Committee of the Erasmus School of Health Policy & Management assessed and waived ethical approval for this study (reference: ETH2122-0605).

We constructed a Bayesian D-efficient DCE with informed priors using Spotlight software [29]. We based the priors of the design for the pilot study on the empirical findings discussed in the Introduction section [15, 18, 19, 25], indicating that respondents were more likely to disagree with a negative reimbursement decision when one of the patients affected by the decision was depicted (by means of presenting a picture of an individual patient) and when patients were younger, more severely ill, and gained more health from. We optimized the design for the main study using priors based on the results of the pilot study. Table S1.1 (Supplementary Material S1) presents an overview of the applied priors.

The DCE design consisted of 120 choice tasks, divided into 10 blocks, to one of which respondents were randomly assigned. Respondents completed a series of 12 choice tasks (in random order) in which we explained that two new pharmaceuticals had become available for two groups of 100 patients (labeled patient group A and B). We (randomly) presented a picture of one of the patients for either group A or B and the text “no picture available” for the other group. The picture was consistently positioned left or right (i.e., for patient group A or B, respectively) for individual respondents to reduce the cognitive burden of completing the choice tasks for respondents. The pictures were selected from Microsoft 365 Free Stock Images on the basis of the criteria that they were as similar (e.g., in terms of the facial expressions of the patients) and neutral (e.g., patients were not depicted in a hospital setting) as deemed possible, within as well as between the different age groups of the patients. The age of the patient in the picture corresponded with the age of the respective patient group (i.e., 10, 40, or 70 years) and was randomly selected from a group of 18 pictures (6 pictures per age group) that were equally distributed across sex and ethnicities. Table S1.2 (Supplementary Material S1) presents an overview of the pictures. Note that we did not obtain ethical approval for examining the effect of the sex and ethnicity of the depicted patients on respondents’ disagreement with the negative reimbursement decisions.

The patient groups were further described (in writing) on the basis of their age, disease severity [i.e., health-related quality of life (HRQOL) and life expectancy (LE) before treatment], HRQOL and LE gains from treatment, and the costs of their treatment. The latter amounted to a total of 200,000 euros per patient, which we used to calculate and present the cost-effectiveness (i.e., defined as “costs per healthy life year gained”) of the new pharmaceuticals and the opportunity costs of their reimbursement (i.e., defined as “expenditure avoided, the euros saved can be spent for treating other patients” in accordance with the definition used in a summary report on reimbursement decisions published by the National Health Care Institute (ZIN) in the Netherlands [30]). We explained to respondents that the healthcare budget was insufficient for reimbursing the new pharmaceutical for both patient groups and that policymakers had decided to not reimburse the pharmaceutical for patient group A (or B, randomly selected). We then asked respondents to indicate whether they agreed with the decision of policymakers to not reimburse the pharmaceutical for patient group A (or B) or whether they believed that the pharmaceutical should not be reimbursed for patient group B (or A)—forcing respondents to clearly express their disagreement. Figure 1 presents an example choice task as presented in version A of the questionnaire.

Example DCE task (version A of questionnaire)

Before conducting the analyses, we formulated the hypotheses that respondents would be more likely to disagree with the policymakers’ decision to not reimburse the pharmaceutical for a specific patient group in case we presented a picture of a patient from that patient group and—in line with related empirical research (e.g., [25, 26, 28]—when the patients were relatively young, had low levels of HRQOL and LE before treatment, and large HRQOL and LE gains from treatment. Furthermore, we formulated the hypothesis that the effect of depicting an affected patient would be stronger when the patients were younger, had lower levels of HRQOL and LE before treatment, and larger HRQOL and LE gains from treatment. To our knowledge, evidence on the effect of respondents’ characteristics on their disagreement with negative reimbursement decisions is not yet available, which is why we explored the data without formulating any a priori hypothesis on this association.

We anticipated that (at least some) respondents might disagree with the negative reimbursement decision—regardless of whether this concerned patient group A or B—and would in fact prefer to reimburse the pharmaceuticals for both patient groups. To reduce the likelihood that such preferences influenced our results (e.g., by way of protest answers), we explained to respondents that we appreciated this preference and that they could leave a comment (on this or anything else) after completing the choice tasks in which they were forced to make a choice for not reimbursing the new pharmaceutical for one of the patient groups.

We selected attributes concerning patients’ age, HRQOL and LE before treatment, and HRQOL and LE gains from treatment. This selection was based on the results of an informal review of the empirical literature on public preferences for reimbursing health technologies on the basis of patients’ age, disease severity, and the size and type of health gains discussed in the Introduction section [24,25,26,27,28]. HRQOL was measured on a visual analogue scale (VAS) ranging from 0 “worst health you can imagine” to 100 “best health you can imagine” and LE was measured in years. We determined the range and levels of the attributes on the basis of the criteria [31]: (i) that the differences in levels could be distinguished by respondents and (ii) that the levels aligned with those commonly used in related empirical studies. The latter would enable us to compare our results with those of (at least some of) the reviewed studies. For reasons of clarity, we also presented the healthy life years gained from treatment and the costs of treatment as attributes in the choice tasks, which we calculated on the basis of the levels of the relevant attributes. In Fig. 1, the 12 healthy life years from treatment gained by patients in group A was, for example, calculated as ((20 points HRQOL gains from treatment × 10 LE gains from treatment) + ((20 points HRQOL gains from treatment + 80 points HRQOL before treatment) × 10 LE gains from treatment)) / 100. The costs of 333,000 euros per healthy life year gained was calculated as ((200,000 euros × 100 patients × 10 LE before treatment) + 10 LE gains from treatment) / (12 healthy life years × 100 patients), rounded to the nearest 1000 euros. Note that we excluded these attributes from the analysis to avoid multicollinearity (Table 1).

The questionnaire consisted of three parts. In part one, we informed respondents about the aim of the study and asked them to give consent for using their data for research purposes. We explained that they could withdraw from the study at any moment, at which time their data would be discarded. We then asked respondents about their age, sex, and education level for sampling purposes and about their health insurance premium, HRQOL “today,” and about what they considered best for themselves and the public in the Netherlands in terms of the coverage and premium of the mandatory health insurance (answer options: they should stay the same, increase, or decrease) for sensitizing purposes. We then explicated the task instruction to respondents and introduced them in three steps to the attributes (including the healthy life years gains and opportunity costs of reimbursement), levels, and choice tasks used in the second part of the questionnaire, the clarity of which we assessed on a 7-point Likert scale (ranging from 1 “very unclear” to 7 “very clear”). We included a practice choice task in each step that built up in complexity to the choice tasks that respondents would complete in the DCE.

In part two, we randomly assigned respondents to one of four DCE versions (labelled A–D). The data obtained in versions A and B was used to meet the aim of the current study. The data obtained in versions C and D was used to meet a different aim, on which we report elsewhere [32]. Versions A and B of the questionnaire were identical, except for the way in which we specified the costs of treatment in the choice tasks. In version A, we presented the costs in terms of the cost-effectiveness of the pharmaceuticals (e.g., specified as “330,000 euros per healthy life year gained” for patient group A in Fig. 1). In version B, we presented the costs in terms of the opportunity costs associated with reimbursement (e.g., specified as “costs of 330,000 euros avoided, this amount can be spent on other patients” for patient group A, assuming the same costs as in Fig. 1). In each version, we asked respondents to complete 12 choice tasks, and subsequently, to leave a comment if so desired. In part three, respondents were asked about their sociodemographic characteristics.

Prior to conducting the main study, we conducted a pilot study to pretest the questionnaire [31, 33]. More specifically, we collected pilot data (n = 406) to assess the clarity of the task instruction and of the attributes and levels, and choice tasks on a 7-point Likert scale (ranging from 1 “very unclear” to 7 “very clear”) as presented in the four (i.e., A–D) versions of the questionnaire. Table S1.1 (Supplementary Material S1) presents the number of respondents per questionnaire version. The mean (SD) clarity scores were 5.8 (1.1) for the instruction and 5.5 (1.3) for the attributes, levels, and choice tasks, which was considered satisfactory. As such, we did not modify the questionnaire for the main data collection (n = 1628) and merged the pilot and main data (total sample n = 2034) before conducting the analyses. Note that we used data obtained from a subsample (n = 1008) to meet the aim of the current study (see Sect. 2.4).

Before assessing the effect of depicting one of the patients affected by policymakers’ decision to not reimburse a new pharmaceutical for patient group A or B on respondents’ disagreement with the decision, we calculated the proportions of (in total 12,096) choice tasks in which respondents (dis)agreed with the decision and of respondents who consistently (dis)agreed with the decision in all choice tasks. We used random-intercept logit regression models to analyze the choice data. These models accounted for the likelihood that choices were nested and that some respondents could be more inclined to (dis)agree with negative reimbursement decisions than others, independent of the picture, attributes, and levels presented in the choice tasks. Note that random-intercept models are a specific type of random-effect models where only the intercept is modeled as a random effect, while the coefficients of the predictor variables are modeled as fixed effects. We used a normal distribution for the random intercept and categorical variables for coding the attribute levels and respondent characteristics.

We first ran six models to examine the main and interaction effects of the choice–task characteristics on respondents’ disagreement with policymakers’ decision to not reimburse the new pharmaceutical for one of the patient groups. By running model 1, we estimated the main effects of depicting one of the patients affected by the negative reimbursement decision by means of presenting their picture (0, no; 1, yes) and of the attributes (i.e., patients’ age, their HRQOL and LE before treatment, and their HRQOL and LE gains from treatment) and levels on respondents’ disagreement. By running models 2–6, we successively estimated the interaction effects of depicting one of the affected patients and patients’ age, their HRQOL and LE before treatment, and their HRQOL and LE gains from treatment on respondents’ disagreement. We then ran 7 models to examine the main and interaction effects of the choice–task and respondents’ characteristics on their disagreement with the negative reimbursement decision. By running model 7, we estimated the main effects of depicting one of the patients, the attributes and levels, and respondents’ age, sex, education level, household income (per month, before tax), having children (no/yes), and HRQOL on their disagreement. Finally, by running models 8–13, we successively estimated the interaction effects between depicting an affected patient and the abovementioned respondents’ characteristics on their disagreement, while controlling for the attributes and levels. We furthermore controlled for the presentation of the picture (i.e., positioned left or right for patient group A or B, respectively) and the treatment costs (i.e., presented as cost-effectiveness or opportunity costs in version A or B of the questionnaire, respectively) in all models.

After running models 1–13, we further explored the choice data by estimating the main and interaction effects of depicting one of the affected patients and patients’ age and their HRQOL and LE endpoint after treatment (calculated as ‘HRQOL before treatment + HRQOL gains from treatment’ and ‘patients’ age + LE before treatment + LE gains from treatment’, respectively), as well as the interaction effect between HRQOL and LE gains from treatment on respondents’ disagreement. We assessed the robustness of our results by repeating the analyses excluding respondents who (i) reported a clarity score < 4 for the introduction to the attributes, levels, and choice tasks; (ii) completed the 12 choice tasks in less than 34 s or more than 17.3 min (i.e., the completion times of the quickest and slowest 5.0% of respondents, respectively) on the basis of the distribution of completion times; and (iii) left a comment saying that the pharmaceuticals should actually be reimbursed for neither or both patient groups, which we identified and clustered using inductive coding methods.

We conducted the analyses using Stata 18.0 (Stata Corp LP, College station, Texas).