Introduction

Safety surveillance is crucial and mandatory in all clinical trials involving investigational drugs or vaccines, whether in the clinical development phase or licensed, as per legislation, regulatory authority guidelines, and the study protocol. In contrast to drug trials, where pre-existing illnesses are treated, vaccines are administered to healthy participants to prevent a disease and vaccine trial participants are followed-up for long periods outside the hospital setting. Due to these reasons, there is special emphasis on surveillance of vaccine safety during clinical trials.

The collection of safety data in clinical trials is typically based on participants reporting all adverse events (AE) to the investigator as soon as possible, along with participant interviews during each study visit or during other contact to the study personnel. If a participant reports an AE, the investigator requests medical files from the health care provider(s) for review and provides an individual report of any serious adverse events (SAE) to the trial sponsor with causality evaluation of the investigational product according to the principles set in the guideline for good clinical practice E6(R2).1 Causality assessment is a key factor in identifying serious adverse reactions (SAR; a SAE with assumed causal association with the investigational product) which require more detailed and urgent reporting by the sponsor to the competent regulatory agencies. However, assessing a causal association between an individual event and a drug exposure is not a standard medical practice and often proves to be difficult and subjective. The interrater agreement in causality assessment methods in pharmacovigilance has varied from fair to moderate.2,3

In the above-described typical setting, the potential for participant recall bias and delays is evident and AEs may also be missed, especially if a clear suspicion of causality between the event (like trauma) and the investigational product is not raised. During a vaccine trial, detection of AEs should take place in real-time and be as comprehensive as possible to safeguard the trial participants and to ensure that the safety profile of the vaccine becomes established before licensure. In the post licensure settings, the World Health Organization recommends that individual case assessment of causality in an AE following immunization should consider the existing population level evidence of association in addition to case-specific factors such as the temporal relationship between the immunization and the event.4

Large pragmatic clinical trials are a feasible way to evaluate the effectiveness of influenza vaccines.5,6 These trials also present new possibilities for safety data collection and evaluation. The FinFluHD trial was a large pragmatic influenza vaccine trial conducted in Finland, where all vaccinations and hospitalizations are reported in almost real-time to national health registers, and a personal identification code of each permanent resident allows deterministic data linkage between the registers.7,8 To enable as comprehensive, real-time, objective and resource-effective safety evaluation as possible during the FinFluHD trial, we developed a hybrid safety surveillance approach utilizing both active and passive safety data collection. Our approach included comprehensive real-time collection of diagnoses related to acute inpatient hospitalizations occurring during six months following the trial vaccination and epidemiological analysis of 1811 diagnosis categories. This register-based approach replaced conventional reporting of each SAE individually based on active data collection.

Here, we present the passive register-based SAE surveillance method and describe its real-time implementation and performance.

Materials and Methods The FinFluHD Trial

The FinFluHD trial (QHD00012, NCT04137887) was designed to study the relative effectiveness of a new quadrivalent high-dose influenza vaccine (QIV-HD) compared to a standard-dose influenza vaccine (QIV-SD).7,8 The Finnish Institute for Health and Welfare (THL) conducted the trial as an investigator and Sanofi Pasteur sponsored the trial. FinFluHD was a pragmatic register-based modified double-blind individually randomized clinical trial where 50% of the trial population were randomly assigned to receive QIV-HD, while the remaining 50% received QIV-SD. During the trial conduct, the investigator, the sponsor, the trial participants, and the hospital personnel evaluating the patients, were blinded, ie they had no access to data related to the random assignment of the QIV-HD or QIV-SD. After the trial database was locked, this blinding was removed. At the time of the trial initiation in October 2019, QIV-HD was not licensed in Finland, while QIV-SD was licensed.

The trial participants were older adults (at least 65 years old at enrollment) who lived in predetermined study areas. Figure S1 shows the number of trial participants by geographic location. The outcome data were collected passively through the nation-wide hospital discharge register (Hilmo), which covers all hospitals in Finland.9 Data linkage between the trial participants and the national registers was possible via the Finnish personal identity code (PIC).

The trial planned to recruit 120,000 participants over multiple influenza seasons. Altogether, approximately 33,000 participants were recruited during the first season in 2019–2020. Thereafter, the trial was on hold due to the COVID-19 epidemic and was permanently stopped in April 2022. The trial vaccinations were carried out between November 4 and December 23, 2019. The enrolment and vaccination were conducted in collaboration with the public HCCs, also responsible for the routine National Vaccination Programme (NVP) vaccinations. There were no active follow-up study visits after the enrolment.

Safety was assessed in real-time during the FinFluHD trial based on a combination of active and passive (register-based) reporting. Using conventional active reporting, serious adverse reactions (SAR), AEs of special interest (AESIs; including new onset of Guillain-Barré syndrome, encephalitis and/or myelitis including transverse myelitis, Bell’s palsy, optic neuritis, and brachial neuritis), and fatalities were analyzed as previously described.8 The passive register-based follow-up was utilized for identification of SAEs from national health registers during the trial, based on predetermined objective criteria without active data collection (SAE surveillance), which we describe in the current study.

Ethics statement

The study was conducted in compliance with the International Conference on Harmonization of Technical Requirements for Pharmaceuticals for Human Use, guidelines for Good Clinical Practice, and the Declaration of Helsinki. The trial protocol was approved by the Hospital District of Helsinki and Uusimaa ethical review board (HUS/2176/2019) and the regulatory agency Finnish Medicines Agency (Fimea). Written informed consent was obtained from the participants before any study procedures were performed.

Register-Based SAE Surveillance

The goal of the real-time SAE surveillance was to assess whether the trial intervention was associated with acute hospitalizations due to any of a specific group of diseases or symptoms. The diseases and symptoms were recognized from diagnoses recorded in electronic patient file systems and the diagnoses were further categorized as SAEs. The SAE surveillance was based on an observational cohort design through comparing the incidences of hospitalization with different SAE categories in the trial population, including both QIV-HD and QIV-SD exposed, with the incidences in those exposed to QIV-SD outside the trial in the national vaccination program (NVP) during the same or previous influenza season. Separately, for each predefined SAE category, a higher incidence in the trial population was interpreted as being potentially related to the trial intervention. The analysis consisted of two consecutive steps: (1) signal detection and (2) signal evaluation. Signal detection involved an automated statistical analysis of 1811 SAE categories and included flagging of categories, which had unexpectedly high incidences in the trial population. Signal evaluation involved a case-by-case expert assessment of any flagged category and, when necessary, register data review on individual cases.

Cohorts and Follow-Up Cohorts

The trial cohort consisted of individuals included in the FinFluHD trial, regardless of the random assignment of the QIV-HD or QIV-SD. The parallel comparison cohort consisted of adults aged 65 years or more (65+) who were not part of the trial and received the NVP QIV-SD vaccination in the trial areas between November 4 and December 23, 2019. The historical cohort consisted of adults aged 65+ who received the NVP QIV-SD vaccine in the trial areas during the previous season between November 4 and December 23, 2018. The QIV-SD product used in the NVP during 2018–2019 was the same as the control vaccine in the trial. The individuals in the historical cohort could be included either in the trial or the parallel comparison cohort if they were vaccinated during the study season. We refer to the parallel comparison cohort and the historical cohort together as the reference cohorts. We linked hospitalization data to all cohort members using the PIC.

Follow-up and data collection

The individuals in each cohort were followed up for a maximum of 180 days starting from their vaccination date. If an individual withdrew consent during the study before the 180 days had passed from vaccination, the follow-up ended on the day of the withdrawal. We did not account for death. During the real-time SAE surveillance, safety data were analyzed on bi-weekly reporting dates when data were retrieved from the national registers and analyzed. The first data collection and analysis was done on November 25, 2019, which was 24 days after the first trial vaccination. The last data collection and analysis was done on August 13, 2020. On each reporting date, we utilized cumulative data available thus far, accounting for follow-up until the reporting date. If the reporting date was, for example, January 14, 2020, the data and follow-up of the trial cohort and parallel comparison cohort were utilized until that date, and for the historical cohort until January 14, 2019. Note that due to possible data delays, the available medical history of any individual could change during the real-time analysis. For this reason, we continued the data collection until August. Figure 1 describes the cohorts and the follow-up of the SAE safety surveillance.

Figure 1 Cohorts and follow-up in the FinFluHD SAE safety surveillance. Upper panel (A) The SAE safety surveillance considered older adults vaccinated in the FinFluHD trial areas. The trial cohort consisted of individuals taking part in the FinFluHD trial, half of which were vaccinated with the QIV-HD and others with the QIV-SD. The reference cohorts consisted of individuals vaccinated in the national vaccination program (NVP) with QIV-SD during 2018–2019 or 2019–2020 influenza seasons. Lower panel (B) The individuals in each cohort were followed for 180 days starting from their vaccination date. Data from the historical cohort was utilized as prior information when evaluating the monthly incidences of each SAE category in the parallel comparison cohort. The cumulative incidences between the trial cohort and the parallel comparison cohort were then compared in biweekly analyses.

Data Sources Vaccinations

In Finland, influenza vaccine was offered to all at least 65 years old citizens free of charge by the NVP. NVP influenza vaccines are distributed through the public HCCs and are given during yearly campaigns between November and December. Most vaccinations during the 2019–2020 influenza season were given during December, later than usual.10 Vaccinations given in both the NVP and in the trial (blinded) were recorded in local electronic patient record systems and delivered into the National Vaccination Register (NVR) as part of an automated and standardized daily data collection.11 NVR includes the name of the vaccine and the date of vaccination along with the date of birth, municipality of residence, sex, and the PIC of the vaccinee. The trial enrolment and investigational product data were also separately recorded and collected for the purposes of the trial, but with delays. Therefore, the NVR data were used in the real-time safety surveillance. Of note, the NVR records did not specify the assignment to QIV-HD or QIV-SD treatment arm. During November–December 2019, most NVP vaccinations (90%) were given at HCC visits but some during home visits (8%) or other care contacts (2%). All trial vaccinations took place at HCC visits.

Hospitalizations

Following the standard procedure in Finland, after any cohort members were discharged from hospital care, discharge diagnoses along with information related to the patient and care were collected from the local electronic patient record system and sent to Hilmo in a standardized discharge record format. The diagnoses were recorded using the Finnish version of the international classification of diseases (ICD10) with minor additions to the version published by WHO.12 A discharge record includes a single primary diagnosis, any number of secondary diagnoses, admission and discharge dates, information related to the urgency and nature of care, information on the data provider, and the PIC of the patient.

Accounting for data delays

At the time of our real-time analysis, depending on the hospital unit and the electronic patient file system in use, varying delays from the patient discharge to the registration of the discharge record to Hilmo may have occurred. Such delays may have caused missingness in the real-time analyses. The hospitalizations of the historical cohort were not affected by delays and were therefore not directly comparable to the hospitalizations of the two real-time cohorts. To account for this difference, we down-sampled the records of the historical cohort. Specifically, on each reporting date, a hospitalization for the historical cohort was included in the analysis if its discharge date was at least one year before the latest (real-time) data delivery date of the corresponding data provider.

SAE Categories

We considered unscheduled acute inpatient hospitalizations following an emergency room visit as SAEs. We allowed multiple hospitalizations for the same patient. We collected all recorded ICD-10 diagnosis codes related to such hospitalizations. We excluded cardiorespiratory diagnoses as they were the main primary effectiveness outcomes in the FinFluHD trial, and the diagnoses corresponding to AESI outcomes as they were analyzed separately. Table S1 shows the diagnoses excluded from the SAE surveillance. The ICD-10 coding is hierarchical: each diagnosis valid for SAE analysis belonged to one of 1577 disease categories (hierarchy 2), one of 212 blocks (hierarchy 1), and one of 22 chapters (hierarchy 0). These 1811 codes from hierarchy levels 2, 1 and 0 were considered as the SAE categories of interest. We mapped each recorded diagnosis to these categories and a single diagnosis thus produced three different SAEs. After mapping the diagnoses collected from a hospitalization, any duplicates were discarded. We derived SAEs from all recorded diagnoses in our main analysis, while a secondary analysis derived SAEs from the primary diagnoses only.

Signal Detection

The objective was to evaluate, at the periodic reporting dates, and for each SAE category independently, whether the cumulative incidence of the SAE category in the trial cohort was higher compared to the cumulative incidence in the parallel comparison cohort. At each reporting date , we evaluated the data accumulated thus far. Here, we present an overview of the statistical methods used for signal detection. A more detailed description is provided under title “Supplementary statistical methods for SAE surveillance signal detection” in the Supplement.

Estimation target

Let denote the cumulative incidence rate of SAE outcome e by day in the individuals belonging to the trial cohort, and the cumulative incidence rate in those in the parallel comparison cohort. We used the ratio

to compare the cumulative incidences. Values were considered as evidence of a higher cumulative incidence of e in the trial cohort.

Adjustment

Trial participants differed slightly from the individuals in the reference cohorts, also with respect to other factors than the possible random assignment to the QIV-HD. For example, the trial participants were on average younger, and the proportion of males was higher than in the parallel comparison cohort. Also, the vaccination schedules in the trial and in the NVP were slightly different and the follow-up was thus distributed unevenly in calendar time between the cohorts. To enable the comparison, we used direct standardization to adjust based on the age and sex distributions of the trial cohort. We further standardised both cumulative incidence rates by calendar month.

Estimation

We used a Bayesian approach to estimate the two cumulative incidence rates, by first using separate Gamma/Poisson models to estimate the rates for each SAE category by stratum (by calendar month for both cohorts and additionally by sex and age categories for the parallel comparison cohort). We used data from the historical cohort as prior information when estimating the rates in the parallel comparison cohort. The historical cohort contributed approximately 50% of the information in the estimation. We then used numerical methods to attain samples from the posterior distributions of , and .

Inference and flagging of SAE

On each reporting date, we evaluated each , ie, the probability that the cumulative incidence of SAE e is higher in the trial cohort than in the parallel comparison cohort. Based on a specificity analysis (see below), a category e was flagged if the following two conditions were met: the number of patients in the trial cohort with SAE e was at least three, and the posterior probability that the cumulative incidence rate of e is higher in the trial cohort was at least 0.99, ie . We derived 98% credible intervals (98% CrI) for the cumulative incidence rate and the incidence rate ratio based on the 1% and 99% quantiles of their posterior distributions. This credible interval was chosen out of the convenient relation to the above flagging rule; for this 98% credible interval of the rate ratio, a lower limit greater than one always corresponds to flagging the corresponding SAE category. We derived the expected number of SAE e in the trial cohort based on the posterior distribution of and the follow up time accumulated in the trial cohort.

Signal detection report

The results of each automated signal detection analysis were presented in a report, which was constructed as part of the automation and contained the following information for each SAE outcome: number of patients in the trial cohort, the observed and the expected numbers of events in the trial cohort, the posterior mean and credible interval for , and an indicator whether the outcome was flagged or not. These data were also presented for a secondary analysis, which considered the main diagnoses only. Additionally, the reports contained, by cohort, frequencies of the specific ICD-10 diagnoses related to any flagged SAE, and the numbers of patients with those diagnoses. The reports were constructed mostly biweekly as previously noted in the section “Cohorts and follow-up”. The analyses were implemented in R version 3.6. and the reports were constructed with R Markdown.13,14

Signal Evaluation

Each flagged SAE category was assessed by a blinded SAE surveillance expert group (SSE group) consisting of the investigator’s team statistician and medical expert and the sponsor’s team epidemiologist and safety expert. The chosen methodology of assessment depended on the category and the number of SAEs observed. First, the SSE group studied the data in the signal detection report. The group considered whether there was any biological plausibility of a causal relationship between the SAE category and vaccination, which could exclude SAEs related to eg, physical injuries from further evaluation. If the possibility of a causal relationship was deemed to exist, and the number of events in both the trial cohort and the parallel comparison cohort were larger than five, a statistician could perform additional category-specific regression analyses as agreed by the group. However, the requirements for the additional analyses were not met.

For the plausible causal SAEs with a reasonable number of events in the trial cohort, the medical expert evaluated the register data for the individual cases. The expert noted the age and sex of each case and reviewed the history of care notifications from Hilmo and the Register of Primary Care Visits (AvoHilmo)15 to assess whether the events were incident cases or pre-existing conditions. If the ICD-10 codes of the SAE, or the diagnoses directly related to them, had occurred in the medical history of the participant before the vaccination, they were considered pre-existing conditions. Additionally, the expert evaluated the plausibility of the events being causally related to vaccination based on (i) the temporal relationships between the vaccinations and the events, (ii) the biological plausibility of the vaccination causing the events, and (iii) available literature about the association between vaccination and the SAE category, as considered necessary. The evaluation could be discontinued at any point if there was agreement within the SSE group that the signal was overwhelmingly likely to be a false-positive finding.

If the evaluation would have resulted in a suspicion of a true positive finding, all the cases would have been reported individually to the sponsor with causality evaluation based on patient file data. However, this did not occur.

Other Analyses Post-Surveillance Analyses

After the real-time surveillance and data collection had ended, we evaluated the incidence of acute SAE hospitalization in the trial and parallel comparison cohorts at the study end. Later, we also had access to unblinded data from the trial and further evaluated the finding from the real-time blinded analysis.

Incidence of acute SAE hospitalization

To quantify differences in overall health between the trial cohort and the parallel comparison cohort, we compared the cumulative incidences of acute hospitalization related to any SAE at the study end. We used Poisson regression to estimate the incidence rate ratio (IRR) with (I) adjustment for age group and sex, and (II) further adjustment for the type of care contact at the time of vaccination (HCC vs other). For both models I and II, we report the IRR estimate and its 95% confidence interval (95% CI) based on a normal approximation for the log-scale sampling distribution of the IRR estimator. These Poisson regression analyses did not include data from the historical cohort. Note that, in contrast to other statistical analyses in this study, we used a frequentist approach for estimating the Poisson regression parameters. The numbers of acute SAE hospitalizations were large, so a Bayesian analysis based on non-informative priors would yield similar results.

Unblinded analyses

Unblinding provided an opportunity to evaluate whether the signals detected during the real-time analysis were associated with the QIV-HD vaccine. After unblinding, we thus evaluated data from the trial participants with flagged SAEs. For each flagged SAE category, we compared the frequencies of those exposed to the QIV-HD and QIV-SD, both at the time of the first flag, and at the end of the study. We evaluated the posterior probability that the proportion of those exposed to the QIV-HD is greater than the proportion of those to the QIV-SD (ie, greater than 0.5). We used a uniform prior and a binomial likelihood and evaluated the posterior probability of interest using the pbeta function in the R software. The justification for the uniform prior is that all SAE were a priori equally plausible to occur in both QIV-HD and QIV-SD groups as the SAE categories were based on pre-existing disease classifications.

Specificity of Signal Detection

Prior to the start of the real-time safety surveillance study, we evaluated the expected number of false-positive findings (Type I error) in the signal detection method. We sampled NVP influenza vaccinated adults (65+) living in the FinFluHD trial areas into cohorts and then simulated the SAE safety surveillance based on those cohorts. We performed the simulation for the 2016–2017 and 2017–2018 influenza seasons. During a simulation season, we formed the mock trial cohort by randomly sampling 30,000 individuals from the NVP influenza vaccinated adults. Those not sampled became the parallel comparison cohort. We then formed a historical cohort from those vaccinated during the previous season. We carried out signal detection as previously described, based on the available hospitalization data for the three cohorts, all exposed to the same influenza vaccine (QIV-SD).

For both seasons 2016–2017 and 2017–2018, we repeated the cohort sampling and signal detection 100 times. We used the following parameters for signal detection in each iteration: the minimum number of patients in the trial cohort was two or three, and the minimum posterior probability that the cumulative incidence is higher in the trial cohort was set at 0.95, 0.975 or 0.99. During each iteration and for each parameter combination, we noted the total number of SAE categories, which were flagged on any reporting date. We did not introduce any vaccine adverse effects for the randomly selected trial cohort and so all possible flags produced in the simulation were false-positive findings.

Table S2 shows, for both simulation seasons, the mean and interquartile range (IQR) for the numbers and percentages of false-positive flags corresponding to the (six) parameter combinations used for signal detection. With a probability threshold 0.95 the percentages of flagged SAE categories during the 2016–2017 simulation season were 5.3% (IQR: 4.7–5.8%) and 5.2% (IQR: 4.7–5.6%) with minimum two and three patients, respectively. With a higher probability threshold 0.975, the percentages were 3.5% (IQR: 3.0–3.9%) and 2.7% (IQR: 2.3–3.0%) with minimum two and three patients, respectively. Due to the large number of SAE categories, even with a high probability threshold 0.99 and minimum three patients, the expected number of false flags was still 29 (IQR: 24–33) and 28 (IQR: 23–33) during the 2016–2017 and 2017–2018 simulation seasons, respectively. In order to limit the expected number of false flags, we utilized these latter parameter choices in the real-time SAE surveillance, and the specificity analysis indicated that the false-positive rate (type I error) of our signal detection was 2% for the entire study period.

Results Results from Signal Detection

Ten SAE categories were flagged during the real-time safety surveillance. Of these, six were COVID-19 related. Figure 2 shows the flagged SAE categories on each reporting date. For each flagged category, Table 1 shows the numbers of patients in the trial cohort, the observed and expected numbers of events, and the RRs with 98% credible intervals at the first appearance of the flag.

Table 1 Serious Adverse Event (SAE) Categories Which Were Flagged at Any Reporting Date During the Safety Surveillance

Figure 2 SAE surveillance results. Left: Serious adverse event (SAE) categories which, during any of the approximately biweekly reporting dates, had an unexpectedly high incidence in the study population and were thus flagged in the corresponding safety surveillance report. Figure shows in which reports the flags appeared. Right: overall findings from the blinded and unblinded SAE safety surveillance analyses.

COVID-19 related flags

Flags for categories U00–U49 (Provisional assignment of new diseases of uncertain etiology) and U00–U99 (Codes for special purposes), first occurred on April 13, 2020, both with 8 patients and outcomes. The expected number of events at that time was 1.5 for both and the RRs were approximately 5.2 (98% CrI: 1.5–12) for both. The latter set of codes stayed flagged until the end of surveillance, while the former was only flagged on that one report as it was removed from the official Finnish ICD10 coding and thereafter excluded from our analysis. After the April 13 report, there were four more COVID-19 related flags: U07, U07.1 (COVID-19, confirmed), Z20–Z29 (Persons with potential health hazards related to communicable diseases), Z29 (Need for other prophylactic measures).

Other flags

K86 (Other diseases of pancreas) was the first SAE category flagged during the safety surveillance on February 3, when there were three patients with four events in the trial cohort with that category, and the expected number of events at that time was 0.14 with RR 29.3 (98% CrI: 5.2–561). The K86 was also flagged during subsequent reports until April 27. On March 16 there were two additional flags with K26 (Duodenal ulcer) and R20–R23 (Symptoms and signs involving the skin and subcutaneous tissues), with 5/5 and 6/6 patients/events, respectively, while the expected numbers of events for these categories were 1.07 and 1.8, and RRs were 4.7 (98% CrI: 1.0–13.2) and 3.3 (98% CrI: 1.0–10.1). The R20–R23 was later flagged twice, while the K26 was only flagged during that one report. The S51 (Open wound of forearm) was flagged on May 25, with 3 patients and events, with expected number of events 0.4.

Results from Signal Evaluation

The unexpectedly high incidences of the COVID-19 related outcomes were considered by the SSE to be consequences of the novelty of the disease and thus they were not evaluated further during the SAE surveillance. One flagged SAE category, S51 (Open wound of forearm), was considered biologically implausible to be caused by vaccination and thus was not further evaluated. Three flagged SAE categories, K86 (Other diseases of pancreas), K26 (Duodenal ulcer) and R20–R23 (Symptoms and signs involving the skin and subcutaneous tissues), were further investigated. Each had a low number of events and therefore additional adjusted statistical analyses were not performed. Nevertheless, the statistician analyzed the historical incidence of the three categories by age and sex categories, and the clinical expert evaluated the individual cases of the trial cohort. For all these categories, the evidence supporting a causal relationship between vaccination and the SAEs was considered weak or unclear: K86 had only a single incident case, and for both K26 and R20-R23 the temporal relationships between the vaccinations and the hospitalizations showed great variation. More details of the investigations are included under title “Supplementary results from signal evaluation” in the Supplement.

Results from Other Analysis Incidence of Acute Hospitalization

The trial cohort included 33,093 vaccinated individuals who contributed a total of 16,216 person-years of follow-up during the SAE surveillance. Altogether, 1217 hospitalizations related to any of the SAE categories (SAE hospitalization) occurred in the trial cohort, contributed by 941 separate individuals. At the end of the study, the adjusted cumulative incidence rates of SAE hospitalization in the trial, parallel comparison and historical cohorts were 68.6, 120.4 and 149.7 per 1000 person-years, respectively. Based on the Poisson regression analysis, the sex and age adjusted IRR for SAE hospitalization between the trial cohort and the parallel comparison cohort was 0.63 (95% CI: 0.59–0.67). When further adjusted for the type of vaccination contact (HCC vs other) the IRR was 0.76 (95% CI: 0.71–0.81).

Unblinded Results Post-Surveillance

For the COVID-19 related SAEs, the numbers of patients exposed to QIV-HD in the trial cohort were similar or greater than the numbers of patients exposed to QIV-SD, both at the time of the first flag and at the end of the study (Table 1). For the categories Z20-Z29 and Z29, events were mostly contributed by a single group of patients (Figure S2), and the numbers of QIV-HD/QIV-SD exposed patients were similar with 10/9 and 9/7 patients at the end of the study, respectively. Greater differences in QIV-HD/QIV-SD exposures were observed for categories U00–U99, U07 and U07.1, for which events were contributed by another group of patients. For the category U07 including both confirmed (U07.1) and suspected (U07.2) COVID-19, the numbers of QIV-HD/QIV-SD exposed patients at the end of the study were 14/10, and the posterior probability that the QIV-HD exposure was more likely with that outcome was 0.79. For the non-COVID-19 related flagged SAEs, the number of patients and events were lower among those exposed to QIV-HD than those exposed to QIV-SD, both at the time of the first flag and at the end of the study.

Discussion Key Results and Interpretation

We developed a register-based method to detect and evaluate safety signals related to serious adverse events in real-time during a blinded phase of a pragmatic clinical vaccine trial, utilizing population-level comparison to aid causality assessment. In our application to the FinFluHD influenza trial in Finland, we targeted 1811 serious adverse event categories, which we recognized from registered diagnoses related to acute hospitalizations. Our method provided information, which we evaluated in real-time during the study: we observed 10 event categories with unexpectedly high incidences in the trial population, and three of the categories were plausible concerns according to expert evaluation. We studied these three further by examining both patient and population-level register data, but these examinations did not reveal strong evidence of causality with the trial vaccination. Furthermore, an unblinded post-surveillance analysis revealed less cases associated with the unlicensed QIV-HD trial vaccine compared to the standard QIV-SD vaccine for all three event categories.

Due to the COVID-19 pandemic, the FinFluHD trial was paused after the first season and eventually stopped. The pandemic provided us with positive control outcomes not present in the historical reference cohort. We observed unexpectedly high incidences of six COVID-19 related diagnoses during the real-time surveillance. The unblinded post-surveillance analysis revealed that the number of hospitalizations associated with confirmed or suspected COVID-19 was greater in those exposed to the QIV-HD than in those exposed to the QIV-SD (9 QIV-HD and 5 QIV-SD patients). In another group of patients hospitalized with diagnoses in the block Z20-Z29 (persons with potential health hazards related to communicable diseases), which were very likely COVID-19 related, the proportions of QIV-HD and QIV-SD patients were similar (10 QIV-HD and 9 QIV-SD). The evidence about the association between QIV-HD vaccination and COVID-19 was therefore unclear.

For the active safety surveillance, four reports of suspected serious adverse reactions (SAR) were obtained from health-care professionals or participants. For two of these, the causality was considered possible by the investigator (one hypersensitivity vasculitis in the QIV-HD group and one Bell’s palsy in the QIV-SD group).8 For two additional reports, the causality was considered implausible by the investigator (atrial flutter and gastroenteritis, both in the QIV-HD group). The case with Bell’s palsy was not evaluated in the SAE surveillance as it was an adverse event of special interest and thus excluded. For the other suspected SAR cases, the corresponding SAEs were not flagged in the SAE surveillance signal detection. In an additional unblinded safety analysis comparing the QIV-HD and QIV-SD groups after the study termination, there were no obvious safety concerns in the analysis of primary discharge diagnoses of all hospitalizations. Unbalanced event counts between QIV-HD and QIV-SD groups were observed for hospitalizations related to pain in throat and chest (R07), open wound of head (S01), intracranial injury (S06), all implausible to be related to either QIV-HD or QIV-SD vaccination (Sanofi, data on file).

Limitations and Generalizability

The nation-wide register data collection we utilized is a comprehensive way of collecting all serious adverse events (SAE) which fulfill predetermined objective criteria, but there are also limitations. An earlier study has reported that the sensitivity of capturing safety data from electronic health records depends on the endpoint of interest.16 However, it is likely that our analysis did not miss any relevant proportion of hospitalizations among the trial population by the end of the study. In Finland, specialized care is available, accessible and affordable to all citizens and permanent residents via a government-funded national health insurance. The coverage of the Care Register for Health Care (Hilmo) is excellent, with an estimated 95% of the hospital discharges in Finland identifiable from Hilmo, according to a systematic review conducted in 2012.17 Our criteria for a SAE was an acute hospitalization associated with a valid ICD10 diagnosis. The algorithm which collects only acute hospitalizations from the Hilmo register is not a trivial one, as the information related to acuteness can be included in four different register variables. While we utilized a custom data management solution in our analysis to collect these hospitalizations, a more general approach would be to first transform the register data to a standard format designed for observational epidemiological studies, such as the Observational Medical Outcomes Partnership Common Data Model (OMOP CDM).18 Such an OMOP CDM database, if updated in real-time, could allow for implementation of our approach to vaccine safety surveillance in any country or organization with sufficient individual-level data on vaccinations and hospitalizations.

Another limitation of our approach was that there was considerable data delay for recent hospitalizations during the real-time analysis. The traditional data transfer schedule to Hilmo has been once per year, which still during summer 2019 was the schedule for over 50% of the specialized care notifications.19 By the beginning of the FinFluHD trial in November 2019, the hospitals providing most of the specialized care in Finland had committed to biweekly data deliveries, and the timeliness and quantity of these deliveries were monitored as part of the FinFluHD study. Due to data delays, we may have under-estimated the incidence of the serious adverse events during the real-time analyses. These delays affected the trial- and parallel comparison cohort equally, and we took them into account when utilizing historical data. Therefore, our signal detection was likely not heavily affected by this possible underestimation. Further, we continued the data collection until August 13, 2020, to include all hospitalizations until the end of the 180-day follow-up period. Following the COVID-19 pandemic since 2020, the timeliness of Hilmo has further improved to a scheduled daily data collection for all hospitals, now including admissions in addition to discharges. By the end of 2021, Hilmo can be considered a real-time register with no delay in the majority of records (Figure S3 and Table S3). When considering utilizing our approach in other settings, minimal delays in the available data on hospitalizations is a crucial aspect.

As our trial enrolment data were recorded manually with long delays, we used National Vaccination Register (NVR) data in the real-time safety surveillance, making the vaccination data directly comparable between the trial cohort and the parallel comparison cohort. Vaccinations given during specialized care or by the private health-care sector were not included in NVR at the time of the FinFluHD trial, but most of the older adults received their influenza vaccinations as a part of the NVP from the public sector. The timeliness of NVR is very good but there can be occasional data delivery issues. During 2015 approximately 87% of HCCs delivered their vaccination records within 7 days from vaccination,20 but since 2018 the median delay of NVR has been only one day during most months (Figure S3 and Table S3). An exception were the early months of 2020 when the delay was historically high (see “Supplementary register analyses: record delays in Hilmo and NVR during 2018–2022” in the Supplement for more information). The 2020 delays in the vaccination records started after the enrolment to our safety surveillance cohorts had ended, and we have no reason to believe that the delays affected our analyses. Following the COVID-19 pandemic, most of the largest private sector health-care providers have also started providing vaccination information to NVR. By the end of 2021, NVR can be considered a real-time register covering the vast majority of private and public sector vaccinations administered in Finland.

Vaccination is a self-selected event and therefore the vaccinated population can be different from the unvaccinated one.21 For this reason, we included those who self-selected to influenza vaccination in the NVP in the reference cohorts. It is likely that these cohorts were still not directly comparable with the trial cohort in terms of baseline health status, as those who entered the trial were on average younger and probably also healthier. We observed a clear and strong healthy enrollee effect, similar to a healthy vaccine effect. Instead of the expected 23–33 false positive safety signals, we observed only 10 signals during the study. Even when adjusted for age, the incidence of hospitalization was still much lower in the trial cohort compared to both reference cohorts (parallel and historical). It is unlikely that the large differences in the incidences are explained by the medical effect of the trial vaccine, especially as the main effectiveness outcomes were excluded from our analysis.

We used direct adjustments for age and sex in our signal detection analysis, but the method should be further improved to better deal with the healthy enrollee effect. The participation to the trial required a health facility visit but some of those vaccinated outside the trial (ie those in the reference cohorts) were vaccinated at home. Our post-surveillance analysis indicated that the place of vaccination was a significant indicator for health in the older adults. When additionally adjusting for the place of vaccination, the hospitalization incidence rate ratio between the trial and parallel comparison cohort was 0.76 (95% CI: 0.71–0.81), closer to one than the ratio without this adjustment, 0.63 (95% CI: 0.59–0.67). Nevertheless, it seems that age, sex and place of vaccination are not sufficient to explain all the observed differences in health between the trial cohort and the reference cohorts. Additional research is needed for mitigating the healthy enrollee effect in future applications.

We did not account for loss to follow-up due to death in our analysis which may have led to underestimation of the incidences of serious adverse events, especially in the older age groups, and therefore especially in the reference cohorts. However, the incidence of any SAE hospitalization was actually much greater in the reference cohorts than in the trial population. Our follow-up of 180 days was relatively short and there were only 194 deaths in the trial population, which represented 0.6% of the population. It is thus unlikely that the possible underestimation due to not accounting for death significantly affected our analysis.

Our analysis targeted 1811 SAE categories, and we used probabilistic methods to analyze the accumulated data related to each category at different points in time. This setting can in general cause issues related to multiple testing, where the actual probability of a false-positive finding would not correspond to the nominal probability of the statistical method. In our analysis, we were interested in each of the SAE outcomes independently. Our research question, therefore, was not, “Is the trial vaccination associated with any of the 1811 SAE categories of interest” but rather, for each SAE category: “is the trial vaccine associated with this specific category”. Therefore, there were no multiple comparisons across the different categories. Nevertheless, we had a multiple comparison issue across different time points, as we carried out multiple analyses over time utilizing the accumulated data. We have studied the frequentist properties of our method by simulation (see “Specificity of the signal detection”) and estimated that the false-positive rate (type I error) of our method was 2%. These simulations included the multiple comparisons across time, and therefore we did not need to further adjust our method for multiple comparison.

In our implementation of real-time signal detection, we monitored the cumulative incidence of serious adverse events up to 180 days post-vaccination. Targeting more specific and shorter risk windows following vaccination is typical of post-licensure vaccine safety studies when there is prior data on the possible temporal association between the vaccination and the event. For example, the Vaccine Safety Datalink project in the USA has monitored the safety of influenza vaccinations using risk intervals such as 0–2, 0–7, 1–21, and 1–42 days following vaccination, depending on the adverse event outcome of interest.22,23 Our method targeted the majority of all possible diagnosis categories, and we did not incorporate outcome-specific a priori information. As most of the cohort was vaccinated during the beginning of the real-time safety surveillance, our signal detection method should detect acute vaccine reactions during the beginning of the surveillance. However, if the risk of the adverse event is only elevated during a short time following vaccination, then those signals may fade as the follow-up continues. Data delays could therefore cause us to miss the signal. A possible solution would be to split the individual follow-up into risk intervals, eg 0–7, 8–42, 42+ days following vaccination and to analyze those intervals separately. In this risk interval setting, data delays are less likely to significantly decrease the probability of detecting true safety signals.24 These additional analyses could, however, produce more false-positive safety signals. A possible solution could be a temporal tree statistic approach, which can be used to analyze multiple possible risk intervals, while also taking into account the hierarchical structure of event categories.25,26

Conclusion

Our SAE safety surveillance was based on real-time register data instead of active data collection from the participants, thus minimizing delays and removing patient recall bias, which may be present in active safety surveillance. The passive safety data collection was nation-wide and not limited to patient files within the investigational site. For detection of safety signals, we used predefined objective criteria based on a probabilistic comparison of incidences. This allowed us to focus the time-consuming case evaluations on only those adverse events that were suspected of a causal relationship, making our approach resource effective. Furthermore, our causality assessment of vaccine and serious adverse event relationship during the real-time safety surveillance was not solely based on individual cases but strengthened by population-level comparisons.

In summary, our hybrid safety surveillance for a pragmatic vaccine trial was feasible and provided safety signals, which were evaluated in real-time. Combined with active safety surveillance for suspected serious adverse reactions, cases of deaths and selected AESIs, it provided reassurance as a risk-based safety follow-up for a Phase III/IV trial. The Finnish infrastructure is especially suitable for this kind of methodology and has been further enhanced by real-time reporting to the national health registers.

Abbreviations

AE, Adverse Event; SAE, Serious Adverse Event; SAR, Serious Adverse Reaction; AESI, Adverse Event of Special Interest; QIV-HD, Quadrivalent Influenza Vaccine, High-Dose; QIH-SD Quadrivalent Influenza Vaccine, Standard-Dose; NVP, National Vaccination Programme; WHO, World Health Organization; THL, The Finnish Institute for Health and Welfare; PIC, the Finnish Personal Identification Code; HCC, Public Health Care Center; NVR, National Vaccination Register; Hilmo, Care Register for Health Care; ICD10, International Classification of Diseases, 10th edition; SSE group, Serious adverse event Surveillance Expert group.

Data Sharing Statement

Individual participant register data can be requested through the national Findata process, see the findata.fi website. Aggregate biweekly safety data reports related to this study can be made available based on a reasonable request to the corresponding author.

Acknowledgments

We would like to thank Sonja Banga and Eng-Soon Chan from Sanofi for their help with MACE and ICD10 terms. We would also like to thank Esa Ruokokoski and Ulrike Baum from the Finnish Institute for Health and Welfare for their help with data management and contributions to safety surveillance methods.

Funding

This study was funded by Sanofi.

Disclosure

TMM and SP are employees of Sanofi and hold stock in the company. TAN, AAP, RKS, HN, and JJ are investigators at the Finnish Institute for Health and Welfare (THL), which received research funding from Sanofi for the current study and from GlaxoSmithKline SA, and Pfizer, Inc. outside of the submitted work. AAP, RKS, HN are currently employed by FVR – Finnish Vaccine Research, which conducts vaccine research funded by a number of vaccine manufacturers, including Sanofi, GlaxoSmithKline SA, Pfizer, Inc., MSD, Moderna, Seqirus and Janssen. AAP has participated in advisory boards for MSD, Pfizer Inc., Janssen, Bionet and GlaxoSmithKline SA, but has received no personal remuneration. RA attended a scientific conference on an invitation from Tillotts Pharma received by her employer HUS. SK and KA report no conflicts of interest in this work.

References

1. European Medicines Agency. Guideline for good clinical practice E6(R2). European Medicines Agency 2016. Available from: https://www.ema.europa.eu/en/documents/scientific-guideline/ich-e-6-r2-guideline-good-clinical-practice-step-5_en.pdf. Accessed Oct15, 2024.

2. Davies EC, Rowe PH, James S, et al. An investigation of disagreement in causality assessment of adverse drug reactions. Pharm Med. 2011;25:17–24. doi:10.1007/BF03256843

3. Thaker SJ, Sinha RS, Gogtay NJ, Thatte UM. Evaluation of inter-rater agreement between three causality assessment methods used in pharmacovigilance. J Pharmacol Pharmacother. 2016;7(1):31–33. doi:10.4103/0976-500X.179361

4. WHO pharmacovigilance team. Causality assessment of an adverse event following immunization (AEFI): user manual for the revised WHO classification, 2nd ed. 2019 update, 9. Available from: www.who.int/publications/i/item/9789241516990. Accessed June16, 2024.

5. Nealon J, Modin D, Ghosh RE, et al. The feasibility of pragmatic influenza vaccine randomized controlled real-world trials in Denmark and England. Npj Vaccines. 2022;7:25. doi:10.1038/s41541-022-00444-6

6. Johansen ND, Modin D, Nealon J, et al. A pragmatic randomized feasibility trial of influenza vaccines. NEJM Evid. 2023;2(2). doi:10.1056/EVIDoa2200206

7. Hollingsworth R, Palmu A, Pepin S, et al. Effectiveness of the quadrivalent high-dose influenza vaccine for prevention of cardiovascular and respiratory events in people aged 65 years and above: rationale and design of a real-world pragmatic randomized clinical trial. Am Heart J. 2021;237:54–61. doi:10.1016/j.ahj.2021.03.007

8. Palmu AA, Pepin S, Syrjänen RK, et al. High-dose quadrivalent influenza vaccine for prevention of cardiovascular and respiratory hospitalizations in older adults. Influenza Other Respi Viruses. 2024:18. doi:10.1111/irv.13270

9. Care register for health care. Available from: https://thl.fi/en/statistics-and-data/data-and-services/register-descriptions/care-register-for-health-care. Accessed June16, 2024.

10. Finnish National Institute for Health and Welfare (THL). Annetut influenssarokotukset kaudella 2021–20, 2021 [Influenza vaccinations administered in the 2021–20 season, 2021]. Available from: https://thl.fi/fi/web/infektiotaudit-ja-rokotukset/rokotteet-a-o/influenssarokote/annetut-influenssarokotukset-kaudella-2021-20. Accessed December2, 2021. French.

11. Baum U, Sundman J, Jääskeläinen S, et al. Establishing and maintaining the national vaccination register in Finland. Euro Surveill. 2017;22(17):pii=30520. doi:10.2807/1560-7917.ES.2017.22.17.30520

12. Finnish institute for health and welfare (THL). Tautiluokitus ICD10, 2011. Available from: http://urn.fi/URN:NBN:fi-fe201205085423. Accessed November29, 2024.

13. R Core Team. R: a language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing; 2019. Available from: https://www.R-project.org/. Accessed November29, 2024.

14. Allaire J, Xie Y, Dervieux C, et al. rmarkdown: dynamic documents for R. R package version 2.0. Available from: https://github.com/rstudio/rmarkdown. Accessed November29, 2024.

15. Register of primary care visits. Availble from: https://thl.fi/en/statistics-and-data/data-and-services/register-descriptions/register-of-primary-health-care-visits. Accessed June16, 2024.

16. Huang Y, Fried LF, Kyriakides TC, et al. Automated safety event monitoring using electronic medical records in a clinical trial setting: validation study using the VA NEPHRON-D trial. Clin Trials. 2019;16(1):81–89. doi:10.1177/1740774518813121

17. Sund R. Quality of the Finnish hospital discharge register: a systematic review. Scand J Public Health. 2012;40(6):505–515. doi:10.1177/1403494812456637

18. Observational Health Data Sciences and Informatics (OHDSI). (2024). The OMOP common data model. Available from: https://www.ohdsi.org/data-standardization/. Accessed October29, 2024.

19. Häkkinen P, Mölläri K, Saukkonen SM, Väyrynen R, Mielikäinen L, Järvelin J. Sosiaali- ja terveydenhuollon hoitoilmoitus 2020 [Social and health care care notification 2020]. Available from: http://urn.fi/URN:ISBN:978-952-343-346-5. Accessed October29, 2024