Figure 1 presents the full DAG which consists of 37 variables. Nine variables (blue nodes) depict the background risk factors for vaccine recipients. Six variables (green nodes) depict key events initiated in the health system (e.g., the distribution of vaccines). The spectrum of expected AEFI is depicted by seven variables (pink nodes), varying in expected frequency from common to rare (e.g., 16% for fever13 and 2.7 events out of 100,000 persons for chest pain14, following vaccination with BNT162b2). Background variables and AEFI together drive the vaccine recipient’s perceived seriousness of an AEFI, which subsequently drives one’s overall level of concern regarding an AEFI. The level of concern can be further influenced by a recipient’s demographic background, which vaccine they received, and any contemporary factors that increase community concerns about vaccination in general, or the particular vaccine (e.g., news about vaccine safety issues). The AEFI, its perceived seriousness, and the recipient’s level of concern, together drive the recipient’s actions including whether they seek medical attention and/or report the AEFI if surveyed. There are 11 actions (purple nodes) and 2 temporary factors (yellow nodes) modelled in the full DAG to illustrate the problem domain. See Supplementary Table 1 for the definition of each variable and a detailed description of the DAG structure. At a high level, there are three main processes that collectively culminate in the ascertainment of an MA report: (1) the vaccine reaction (biological process), (2) the seeking of medical attention (behavioural process), and (3) the data capture process (procedural process). There are many potential interactions among these processes. The key variables extracted to form the simplified DAG are highlighted in blue text.

The figure depicts how the AusVaxSafety active surveillance system for short-term AEFI monitoring operates as a complex system. See Supplementary Table 1 for the definition of each variable and a detailed description of the DAG structure.

The simplified DAG (Fig. 2) contains the key variables that we considered to adequately capture the full causal model for the purpose of subsequent investigations, namely the person’s age at vaccination (age), vaccine received (vaccine), whether they experience reaction within 3 days following vaccination (reaction), whether they seek medical attention (MA), whether they respond to, or participate in the survey (survey participation), and whether they report seeking medical attention (report MA). These represent a minimum set of variables required to investigate how distinct survey response behaviour in subgroups (e.g., different age groups) can interact with the underlying biological process of interest (vaccine reaction, MA) to affect the sensitivity and specificity of the PPA method when applied to observational data (reporting MA via survey participation). In the next section ‘Data simulation and scenario design’, we describe the design and underlying rationale for each scenario and the assumed data generation process. We assumed that the method of survey collection is constant and external influences are time invariant.

The figure depicts the relationship of a minimum set of variables required to investigate how biological, behavioural and procedural processes can interact to influence the probability of a reported MA using PPA method. See Table 1 for the definition of each variable and see Supplementary Information: Section A for how each variable is aligned with the full DAG.

We simulated the five variables represented in the simplified DAG using eight parameters as defined in Table 1. The vaccine variable was not explicitly included in the statistical model as a single type of hypothetical vaccine was considered. For survey responders, only age and reported MA can be observed and the other factors are not observed. In the simulations, these parameters and their values were chosen based on several key assumptions. In brief, we assumed that the more severe an AEFI, the more likely a person is to seek medical attention and to respond to the survey. Compared to older people, younger people are assumed to be more likely to have a moderate or severe reaction13,15, but less likely to respond to a health survey16,17, and less likely to seek medical attention if they experience an AEFI18. We sampled age from a truncated Gaussian distribution with a mean of 43.5 years and a standard deviation of 18.6 years, informed by the age distribution of vaccine recipients observed by AusVaxSafety during the COVID-19 vaccine roll out in Australia. Table 1 defines variables in the simplified DAG, with parameters designed to generate data for each variable using Monte Carlo methods. Parameter values are chosen to describe a scenario for a relatively low prevalence of moderate to severe reaction, low survey participation and a weak influence of moderate to severe reaction on both survey participation and MA. We subsequently used this as a Reference Scenario for PPA investigation. See Supplementary Table 3 for further details about how these parameters were used to generate the event probabilities in the data simulations, including the probability of a participant reporting MA.

To facilitate the investigation of how combinations of these factors influence signal detection, we simulated data for the Reference Scenario consisting of 50,000 hypothetical vaccine recipients representing accumulated safety data to date, and 12 Investigation Scenarios consisting of 4000 hypothetical vaccine recipients representing a typical number of surveys issued to vaccine recipients in a 2-week investigation period. In contrast with the Reference Scenario of a relatively low probability of moderate to severe reaction (Low R), low survey participation (Low P), and a weak influence of reaction on survey participation and MA (Weak), we altered the value of four parameters outlined in Table 2 for each Investigation Scenario. These include the probability of moderate to severe reaction (θ) from a relatively low (Scenarios 1–6, Low R) to a relatively high (Scenarios 7–12, High R) arbitrarily, in other words, under the High R Scenarios there’s a true change in the biological process compared with the Reference Scenario, i.e., a true increase in the risk of moderate to severe reaction as may plausibly occur due to a manufacturing issue with a particular vaccine batch. We varied the probability of survey participation (η) representing a change in the behaviour of vaccine recipients (Medium P for Scenarios 2, 5, 8, 11 and High P for Scenarios 3, 6, 9, 12). Finally, we varied the influence of reaction on both the probability of survey participation (τsp) and the probability of MA (τma) (Strong influence for Scenarios 4–6, 10–12).

We paired the Reference Scenario with each Investigation Scenario, and assessed how likely the PPA method is likely to detect a signal under each Investigation Scenario. We inspected the diagnostics, using the summary function in the cmdstanr package in R, for the first 5 simulations for each of the 12 scenarios. All R-hat convergence diagnostic values were <1.01 and the bulk and expected sample sizes were sufficiently large (>1300). There was no evidence of nonconvergence of any chain. We present a histogram of where each simulation’s number of MAs under the Investigation Scenario sits as a percentile of the predicted probability distribution in Figs. 3 and 4. We also presented here the percentage of simulations that generated a signal for each scenario by age group in light blue and dark blue text.

Signals are flagged when the number of MAs exceeds the 99th percentile (dotted lines) upon 5000 simulations with 4000 individuals simulated for each simulation per scenario. The proportion of simulations which generated a signal within each scenario is represented as percentages within each panel in this figure. The probability of survey participation (η) was varied from Low P (Scenarios 1, 4) to Medium P (Scenarios 2, 5) and High P (Scenarios 3, 6). The influence of reaction (τsp and τma) on both survey participation and MA respectively was varied between Weak for Scenarios 1–3 and Strong for Scenarios 4–6. The exact parameters altered for each scenario are as specified in Table 2, all other parameters used are as specified in Table 1.

Signals are flagged when the number of MAs exceeds the 99th percentile (dotted lines) upon 5000 simulations with 4000 individuals simulated for each simulation per scenario. The proportion of simulations which generated a signal within each scenario is represented as percentages within each panel in this figure. The probability of survey participation (η) was varied from Low P (Scenarios 7, 10) to Medium P (Scenarios 8, 11) and High P (Scenarios 9, 12). The influence of reaction (τsp and τma) on both survey participation and MA, respectively, was varied between Weak for Scenarios 7–9 and Strong for Scenarios 10–12. The exact parameters altered for each scenario are as specified in Table 2, all other parameters used are as specified in Table 1.

Figure 3 consists of the six Low R Investigation Scenarios where there is no change in the biological processes of interest compared with the Reference Scenario, and Fig. 4 consists of those High R Investigation Scenarios where there is an increase compared with the Reference Scenario. It is desirable for the PPA signal detection method to generate a safety signal in investigation scenarios that are set with a high prevalence (probability) of moderate to severe reaction (Fig. 4) and not to generate a signal in scenarios set with a low prevalence of moderate to severe reaction (Fig. 3). Any signals generated in simulations of high and low prevalence of reaction scenarios were therefore considered to be appropriate and inappropriate, respectively.

In Low R Scenarios (Fig. 3), signals were more likely to be inappropriately generated in the older age group when the influence of reactions on survey participation and MA was strong (27–78% of simulations, Scenarios 4–6). This is because the inflated proportion of reported MA is more likely to be detected when the number of survey participants increases due to increasing survey participation and healthcare-seeking behaviour. This same analogy applies to the relatively High R Scenarios (Scenarios 10–12). The variation in the proportion of signals detected across the scenarios is due to a combination of the precision of the posterior predictive distribution and magnitude of the bias, both related to the survey participation rate. Bias in the estimate of reported MA in the Strong Influence Scenarios (Scenarios 4–6 and 10–12) is greatest when the survey participation is low, but the precision of the posterior predictive distribution will also be lower. The absolute inflation in the proportion of respondents who report MA was higher in the older age group compared to the younger age group across all scenarios. For the younger age group, only a low proportion of simulations resulted in an inappropriate signal generation (0–4% of simulations).

In High R Scenarios, the probability of appropriate signal generation was lower (1–62% of simulations) in the younger age group, and in the older age group when the influence of reactions on survey participation and MA was weak (12–43% of simulations).