Study sample

We used Prolific and its filtering options to collect complete data from 1200 U.S. adults and obtain a relatively balanced sample of participants with anti-vaccination, neutral, and pro-vaccination attitudes. We created three instances of the same study, each available to 400 participants. One was run with participants with anti-vaccination attitudes, another with participants with pro-vaccination attitudes, and a third with participants with neutral vaccine attitudes, as declared in the survey that Prolific provides to all platform users. At the end of the study, we asked participants about their vaccination attitudes to check for possible changes since the initial measurement by Prolific. Specifically, we asked “How would you characterize your general attitude toward vaccination against COVID-19? [Against, Neutral, Pro].” Most observed changes were in the pro-vaccination direction, and the final sample consisted of 365 anti-vaccination, 373 neutral, and 462 pro-vaccination participants. For all our analyses, we used the attitude that the participants reported at the end of the study.

The sample consisted of 720 women (60%), 463 men (39%), and 17 participants who chose “other” as their gender. The mean age was 38.23 years (SD = 13.76). We estimated that participants would need up to 20 minutes to complete the study; the actual median completion time was around 12 minutes. Participants were remunerated with 2.50 GBP. See Supplementary Fig. 1 for the distribution of all collected demographic variables within each attitude group.

The study was approved by the Internal Review Board (Ethics Committee) of the Max Planck Institute for Human Development. To participate in the study, each participant had to provide informed consent by accepting the terms and conditions outlined in the Study Information and Statement of Informed Consent for Adult Participants, presented at the beginning of the procedure. Data collection occurred between April 19 and April 25, 2022.

Vaccine selection

We included eight existing COVID-19 vaccines in the decision task, asking participants to state their willingness to accept or refuse vaccines (see Fig. 2 and “Methods: Mouselab task”). There were several reasons why we included more vaccines than only the three available in the U.S. First, we included multiple vaccines to be able to obtain a more precise measurement of individual tendencies to refuse or accept a vaccine. Second, the different vaccines differed in terms of their worst side effect and their effectiveness statistics (see Table 1); including these vaccines allowed us to get more precise estimates of the tested effects of probability neglect, loss aversion, and probability weighting. Third, we considered it relevant to include vaccines that are based on different technologies and/or were developed in different countries, as we wanted to measure to what extent participants’ decisions were sensitive to factors other than vaccination risks and benefits. Finally, including vaccines not available in the US reduced the possibility that people already had extensive knowledge of the vaccines’ risks and benefits.

Table 1 Data presented in the Mouselab task

Our specific inclusion criteria were as follows: We included all vaccines for which reliable clinical trial data of levels 3 and 4 were available in English at the time the study was designed. Initially, we also planned to include the Sputnik vaccine, but it was dropped when Russia launched the full-scale invasion of Ukraine.

Vaccine evidence data

The vaccine evidence information (i.e., vaccine effectiveness, side effects, and the corresponding probabilities) about the eight vaccines that were presented to the participants in the mouse lab choice is provided in Table 1. A list of the data sources we used was sent to the participants after they had completed the study; the list can be obtained upon request. In general, we drew on phase 4 trials (i.e., data collected from monitoring the vaccine after releasing it to the public) for the data on vaccines’ effectiveness. If phase 4 trials were unavailable, we used results from phase 3 clinical trials (i.e., double-blind clinical trials involving thousands or tens of thousands of participants). The main sources for the side effects and their frequencies were also results from phase 3 trials or government reports. The latter were used mainly to obtain the frequencies of extremely rare side effects. We selected the three most severe side effects for each vaccine listed in the sources.

The eight selected vaccines included 15 side effects differing in severity: (1) Mild side effects: fever, tiredness, headache, and muscle pain; (2) Severe side effects: severe general discomfort, severe drowsiness, severe tiredness, severe headache, and severe muscle pain; (3) Extreme side effects: blood clots (thrombosis with thrombocytopenia syndrome), immune system attacking the nerves (Guillain-Barré syndrome) or the blood (immune thrombocytopenia), facial paralysis, heart muscle inflammation (myocarditis), and heart membrane inflammation (pericarditis).

Experimental design

All participants explicitly consented to the conditions of the study. The study had four main parts: (1) a Mouselab task, (2) a willingness-to-pay task implemented for exploratory analyses and not reported here, (3) an affect rating task, and (4) a post-experimental survey. Each task began with a brief introduction of its general purpose. The procedure was programmed using JavaScript and JSpsych by the first author and Maik Messerschmidt from the research IT support team of the Center for Adaptive Rationality at the Max Planck Institute for Human Development. The procedure, including the consent form, is available in a preview mode at https://covid-vax.exp.arc.mpib.org/.

Mouselab task

Participants were presented with information on eight existing COVID-19 vaccines, one after the other, and were asked to indicate for each vaccine whether they would be willing to get vaccinated with it. The task started with a general introduction followed by detailed explanations of the task and the vaccine-related information, and a tutorial consisting of an example decision in which participants could try out how the information inspection boxes work. The vaccine-related information consisted of the developer/brand, country of origin, vaccine technology, risks (side effects and their frequencies), and benefits (effectiveness of the vaccine against COVID-19 infection, severe illness, and death due to the disease).

In each trial, the vaccine brand, country of origin, and vaccine technology were visible at the top of the screen. The outcomes of the vaccine (i.e., side effects and benefits) and their probabilities were hidden behind labeled black boxes; this information could be revealed by moving the mouse cursor over the box (Fig. 2). The information was visible as long as the cursor hovered over the box. Participants could freely explore the information as long and as often as they wished before making a decision, and the program recorded each hovering event. To approximate how vaccination risks and benefits tend to be presented in real life, the probabilities of side effects were presented as number of cases per 1,000,000 people, and effectiveness was presented using percentages that designated the relative risk reduction in the vaccinated population relative to the unvaccinated population. The presentation order of the vaccines, the relative position of risks to benefits, the relative positions of probabilities to outcomes, and the yes/no buttons were randomized between participants. However, to avoid confusion, the relative position of probabilities to outcomes and the yes/no buttons were held constant for each participant.

Before the actual task, participants were presented with a statement on the reliability of the presented data and were asked to “assume that the figures presented refer to the current wave of the pandemic and apply to you personally”: Please keep in mind that the figures provided for the vaccines were taken from official sources (vaccine package leaflets, clinical trial reports, government reports) and reflect the best current state of knowledge. However, as the pandemic evolves, these figures may change, especially as new variants emerge. In addition, data may vary across countries, age groups, and health conditions, and due to other factors. It is therefore possible that the figures presented here deviate from those you may have encountered in other contexts. This is unavoidable, but it does not mean that the figures presented here are incorrect. For the purpose of the study, please assume that the figures presented refer to the current wave of the pandemic and apply to you personally.

Affect rating task

This task consisted of two parts: affect ratings of the potential risks of the vaccines and affect ratings of the potential benefits. The presentation order of these parts was randomized between participants. Participants were asked to rate the overall negative and positive affect associated with each side effect and benefit.

The instruction for rating the side effects was as follows: In this task, you will be presented with a list of possible side effects. Your task is to imagine experiencing each of them after a COVID-19 vaccination. Please indicate the amount of negative emotion you would feel as a result of experiencing the event. We mean any negative emotion, such as feeling distressed, upset, guilty, ashamed, hostile, irritated, nervous, jittery, scared, or afraid.

Participants were then shown a Likert matrix table with 15 rows, each corresponding to one of the side effects (see Table 1).

The instruction for rating the vaccination benefits was as follows: In this task, you will be presented with a list of the negative outcomes that vaccines protect against. Your task is to imagine that you are fully protected from each outcome. Please assess the amount of positive emotion you would feel as a result of being protected from the event. We mean any positive emotion, such as feeling excited, enthusiastic, proud, determined, relieved, strong, or active.

Participants were then shown a Likert matrix table with three rows, each corresponding to one of the benefits (see Table 1).

The rating scale and example emotions listed in the instructions were based on the PANAS scale54. The labels for the negative and positive scales were identical. In both parts of this task, the order of the outcomes in the rating matrices was randomized for each participant.

Post-experimental survey

At the end of the study, we collected the following demographic information: sex (male, female, other), age in years (open-ended), racial identity (White, Black, Asian, multiracial, other), education (≤high school, some college education, Bachelor’s degree, ≥Master’s degree), political orientation (Democrat, Republican, Independent, other), and annual income (0–$30,000, $30,001–$60,000, $60,001–$99,999, ≥$100,000). Participants then reported the number of COVID-19 vaccinations they had received (open-ended), which vaccine they had received (BioNTech/Pfizer, Johnson & Johnson, Moderna), and how many times they had tested positive for COVID-19 (open-ended). We also asked: In your opinion, how likely is it that in the future you will (get COVID-19, get severe COVID-19, die from COVID-19). Participants answered this question using a rating scale with four options: definitely not, not likely, somewhat likely, and very likely. To measure vaccination attitude, we asked: How would you characterize your general attitude towards vaccination against COVID-19? (neutral, pro, against). The final question was open-ended: Were there specific reasons for how you searched for information about the vaccines in the decision task? How would you characterize your search behavior?

Preprocessing of information inspection data

An instance of information inspection was defined as an event during which a participant hovered a mouse cursor over a labeled black box (Fig. 2a). Following standard practice, inspections that lasted less than 200 milliseconds were assumed to be incidental and removed from further analyses55. The remaining inspection data was used to construct trial-level (i.e., relating to a single vaccination decision) indices of deliberate ignorance for later usage in statistical and computational modeling. We based our analyses on the number of inspections of each piece of information rather than on total inspection times. The information we presented varied in format and character lengths (e.g., frequencies, percentages, text), which could affect inspection duration.

We distinguished between three levels of deliberate ignorance: full, partial, and none. The level of full deliberate ignorance was assigned to trials in the decision task in which no information on vaccine evidence was inspected. The level of partial deliberate ignorance was assigned to trials in which at least one information box on vaccine evidence was uncovered, excluding trials in which all information was inspected. The level of no deliberate ignorance was assigned to trials in which each piece of information on vaccine evidence was inspected at least once.

For all types of probability neglect investigated in the analyses (i.e., for benefits, side effects, and extreme, severe, and mild side effects), we distinguished between two levels: probability neglect either occurred or did not occur in the information inspection phase prior to the vaccination decision. A trial was classified as involving probability neglect of side effects if probability information for at least one side effect was not inspected, but the corresponding side effect was inspected; the same logic was used for benefits and for specific groups of side effects (i.e., mild, serve, or extreme).

Statistical modeling

All statistical models presented were estimated using the brms package56 called from R57. All predictors were categorical and always coded with sum-to-zero contrasts. Posterior distributions of the models were estimated using four chains. Each chain consisted of 4000 iterations. The first half was used for burn-in, and only every second sample was recorded from the second half, resulting in 4000 recorded samples in total. The sampling procedure resulted in well-mixed chains, as indicated by \(\hat\) values lower than 1.01.

We ran Bayesian hierarchical logistic regressions with a random intercept across participants for binary outcome variables (i.e., vaccine acceptance and probability neglect). For ordinal outcome variables (i.e., deliberate ignorance and affect ratings), we used Bayesian hierarchical ordinal regressions, developed specifically for these types of variables58. As priors for the regression coefficients in both types of models, we used zero-centered Student’s t-distribution, with a scale parameter of 2.5 and 3 degrees of freedom, which is considered a weakly informative prior (see: https://github.com/stan-dev/stan/wiki/Prior-Choice-Recommendations).

The models were able to adequately capture the patterns in the data, as indicated by posterior predictive checks59. The approximated out-of-sample predictive performance of the reported statistical models varied from. 7 to. 87 (see Supplementary Information for more details on the evaluation of statistical models).

Predicted outcome values and pairwise comparisons

The posterior predicted outcome values presented in Figs. 3–6 were calculated using the conditional_effects function from the brms package used to estimate the models56. The predictions for a given predictor from a regression with multiple predictors were derived by setting all other predictor values to zero. Because all our models contained categorical predictors coded with sum-to-zero contrasts, these predictions are equivalent to taking the posterior of the global intercept from the model and adding it to the posteriors of the regression weights of a predictor of interest and passing the resulting values through the relevant link function (e.g., the inverse-logit function in the case of logistic regression).

To compute evidence for a difference between any two levels of a categorical predictor (reversed blue triangles in Figs. 4–6), we also drew on the posterior distributions of the regression weights. For a predictor with only two levels (e.g., attentional probability neglect: yes vs. no), we inferred that the data provided evidence for a difference in the outcome variable if the 95% HDI of the posterior distribution of the regression weight excluded zero.

For categorical predictors with more than two levels, the procedure for pairwise comparisons was more complex due to the sum-to-zero contrast factor coding used to estimate the models. First, for each factor level (e.g., vaccination attitude groups), the posterior predicted outcome value on the scale of the linear predictor was computed using the respective factor coding scheme and regression weights. Second, these posterior predicted outcome values were subtracted from each other to derive the posterior distribution of the outcome difference between any two levels of a factor of interest (e.g., anti-vaccination and neutral attitudes). Again, we inferred that the data provided evidence for a difference in the outcome variable between the two levels of a factor if the 95% HDI of the posterior distribution of the regression weight excluded zero.

Computational modeling

The model was written in the Stan programming language for statistical computing60. The posterior distribution of the model parameters was estimated using the rstan package61 called from R57. The sampling procedure from the posterior distribution was based on four chains, each consisting of 2000 warm-up and 3000 subsequent samples. Every other sample was recorded, providing 6000 recorded samples in total. The sampling procedure resulted in autocorrelation-free and well-mixed chains, as indicated by \(\hat\) values lower than 1.01.

The leave-one-out balanced predictive accuracy for the models reported in the main text was 0.81, 0.74, and 0.76 for the anti-vaccination, neutral, and pro-vaccination groups, respectively. The Supplementary Information provides additional analyses of model performance evaluation, including a comparison of the performance of alternative, simpler models.

Formal model specification

The probability of individual i accepting the vaccine v, denoted by P(accept), was given by

$$P(\,\text\,)\,=\frac_+___+\varphi _)}.$$

(2)

The βi parameter represents an individual-level decision bias: It indicates to what extent the participant i tends to accept or refuse a vaccine irrespective of the vaccine properties and evidence. The term Xvβj consists of a 3 × 4 matrix of sum-to-zero contrasts Xv and corresponding βj parameters. The first two columns of Xv code the country of origin (United States, China, other); the next two columns code the vaccine technology (mRNA, vector, other). The term ϕVi,v is the participant’s i subjective value of vaccine v determined based on prospect theory32. The Vi,v component consists of the value function v(a), which takes participant i’s affect ratings for side effects ai,se and benefits ai,b as inputs, and of a probability weighting function w(p), which takes the probabilities of side effects pse and benefits pb as inputs (with the latter technically being the effectiveness of the vaccine, see Supplementary Information):

$$_=\sum _v(_)w(_)+\sum _v(_)w(_).$$

(3)

The value function v(a) has three cases, depending on whether the affect rating pertains to a side effect (ai,se), a benefit (ai,b), or an outcome ignored in the information inspection pre-decision phase:

$$v(a)=\left\-_| a^,\quad &\,\text\,a=_\text\\ (1-_)^,\quad &\,\text\,a=_\text\\ 0,\quad &\,}\,}\,}\,,\end\right.$$

(4)

where the λi∈ [0, 1] parameter is a measure of loss aversion estimated separately for each participant i. With λi = 0.5, both side effects and benefits are weighted equally, while values of λi > 0.5 indicate an overweighting of side effects relative to benefits—which can be interpreted as loss aversion. Note that in most applications to monetary lotteries, the loss aversion parameter Λ is used as a multiplier of the negative consequences and estimated on the scale of positive real numbers. Our approach is algebraically equivalent since Λ = λi/(1 − λi), but resulted in better model convergence.

The α > 0 parameter allows for a nonlinear transformation of the affect ratings. This may be necessary because the affect ratings were measured using an ordinal Likert scale (see Fig. 2b), and the assumption of equal distances between the scale levels may not hold here (but the model allows for a linear mapping with α = 1). Note that when the α parameter is large, for example, when α = 3, the value of an extreme affect coded as a = 5 would be v(5) = λ × 125. Such large values would have a dominating effect within the logit function in Equation (2). For this reason, the Vi,v component is scaled with the φ ∈ [0, 1] parameter—allowing any level of nonlinearity in the affect ratings scale but ensuring that the final value of the φVi,v term is as large as supported by the data.

The function w(p) in Equation (2) transformed the probabilities of side effects and benefits into decision weights and had two cases, depending on whether the probability was inspected or neglected (i.e., deliberately ignored):

$$w(p)=\left\\exp (-^_})\quad &\,}\,\,p\,\,}\,}\,\\ 0.5\quad &\,}\,\,p\,\,}\,}\,.\end\right.$$

(5)

The first case in Equation (5) is an inverse S-shaped probability weighting function62 that transforms objective probabilities into subjective decision weights. The free parameter γi∈ [0, 1] governs the curvature of the probability weighting function and is interpreted as probability sensitivity, with higher values indicating higher sensitivity. When γi = 0, the function becomes a horizontal line with all decision weights w(p) = 0.37. When γi = 1, the function indicates perfect probability sensitivity, that is, w(p) = p. The second case of Equation (5) applies in situations in which an outcome was inspected but its corresponding probability was not; in the main analyses, for these instances of probability neglect, we set w(p) = 0.5, which means that the decision-maker acknowledges the probabilistic nature of the inspected outcome. See the Supplementary Information for an extended discussion on the assumptions underlying the estimation of the weighting function, including the value of the neglected probability and interpretation of the vaccine effectiveness.

In sum, three parameters of the model were estimated on the individual level (i.e., for each participant): decision bias βi, loss aversion λi, and probability sensitivity γi. The parameters α and φ were only estimated on the group level because there were only eight data points (i.e., decisions) per participant, and we had no theoretical interest in estimating these parameters on the individual level.

Stan implementation and prior distributions

The individual-level parameters were modeled as a sum of a corresponding group-level parameter and individual-level displacements ζi:

$$\begin_=\beta +_^\\ _=\Phi (^+_^)\\ _=\Phi (^+_^),\end$$

(6)

where the function Φ() is an approximation to the cumulative normal distribution function implemented in Stan and ensures that the resulting individual-level parameters are always in the required 0–1 range. The individual displacements are assumed to follow a multivariate normal distribution with mean μ = [0, 0, 0] and variance–covariance matrix Σ, also estimated from the data.

In terms of priors, we used standard normal distribution for the individual- and group-level decision biases βi and β, respectively, and also for the vaccine effects coefficients βv, thus assuming that the biases to accept or refuse a vaccine are equally likely.

We could also use the standard normal distribution as the priors for the group-level parameters on the probit scale: λΦ, γΦ, and φΦ, which after transformation to the scale of actual parameter values via probit function Φ−1() resulted in uniform priors on the 0–1 range—in line with the theoretical bounds of the parameters:

$$\begin\varphi =^(^)\\ \lambda =^(^)\\ \gamma =^(^).\\ \end$$

(7)

The α parameter was also modeled on the scale of real values and received a normal prior with a mean of zero and a standard deviation of 0.5. The parameter was then transformed via the exponential function to the scale of positive reals, resulting in a prior with a mode of one (i.e., linear usage of the Likert scale) and assuming that the plausible parameter values are in the 0–4 range.

Finally, to model the multivariate distribution of the individual displacements ζi, we used weakly informative Lewandowski-Kurowicka-Joe (LKJ) prior with parameter η = 5 for the correlation matrix, which assumed that the most probable correlations between the individual parameters were in the range from −0.5 to 0.5. The prior for the standard deviation of the individual displacements \(_^\) was the gamma distribution with shape and rates equal to 2 and 1, respectively, thus ensuring that the parameter is positive and likely in the 0–3 range. The priors for the standard deviations of the individual displacements \(_^\) and \(_^\) were normal distributions with a mean of 0.5 and standard deviation of 0.13, ensuring that the resulting standard deviation is within 0–1 range. This condition was necessary to avoid bimodal individual-level posterior distributions of the λi and γi parameters after the Φ() transformation in Equation (6).

Preregistration

The study design, including sample size and the number of vaccination decisions, was preregistered on April 19, 2022: https://aspredicted.org/66W_95Q. Four research questions were formulated, all relating to probability neglect. Specifically, we were interested in (1) whether people exhibit probability neglect in vaccination decisions, (2) whether probability neglect rates are associated with vaccination attitudes, (3) how probability neglect is associated with vaccination decisions; (4) to what extent probability neglect applies to side effects and benefits. All these research questions are addressed in the main text in the section “How was probability ignorance related to vaccine refusal?” using analytical methods specified in the preregistration. In the preregistration, we also considered weaker definitions of probability neglect (e.g., a lower number of acquisitions of probability information than of the corresponding outcome) than the one used in the paper. However, we decided to use the stricter definition of probability neglect here: acquisition of outcome but not probability information.

At the end of the preregistration, in the section titled Other, we also mentioned our plan to analyze the data using computational modeling and noted that to this end, we would collect affect ratings for side effects and benefits to use as numerical inputs in the models. Our modeling approach followed current best practices (see “Methods: Computational modeling”) and was based on prospect theory—a model previously used to analyze medical choices25,26,27,28,29,63. The model we developed deviated from the standard applications to simple monetary lotteries because vaccination decisions are more complex. Our model was developed to accurately capture patterns in the data and account for various factors driving vaccination decisions (see Supplementary Information).