A specific COVID-19 vaccine-dedicated web-based reporting form [16] was developed that enabled the collection of spontaneously reported information on the vaccine administered, AEFIs and other information needed for assessment and signal detection (Fig. 1). For other medicinal products and vaccines reporting schemes were operated as before. In the COVID-19 vaccine reporting form, patient-friendly descriptions helped in selecting the most appropriate terms, for example, the brand name of the vaccine and a patient-friendly translation of the right Medical Dictionary for Regulatory Activities (MedDRA) term [17], to collect the AEFIs. The most common well-known AEFI that were labelled in the Summary of Product Characteristics (SmPC) at the time of marketing authorization were prespecified on the reporting form to process these ICSRs rapidly. These included injection site reactions, fever, chills, headache, nausea, myalgia, arthralgia, malaise and fatigue. For injection site reactions and fever, conditional questions were asked to gather more detailed and high-quality information to distinguish between the highest body temperature measured or different types of injection site reactions, injection site inflammation and extensive limb swelling.

Structure COVID-19 reporting form

To process answers automatically in specific ICH-E2B(R3) fields [18] in the ICSR management system, specific COVID-19-related closed questions were asked, for example, past drug therapy on previous COVID-19 vaccines and medical history on previous COVID-19 infection(s).

Together with the prespecified AEFIs, most of the fields are filled automatically in our ICSR management system. The data structure of this system is compliant with the ICH-E2B(R3) format, which allows for standardised storage and exchange of ICSRs worldwide. All AEFIs were coded according to MedDRA (versions 23.1, 24 and 24.1) [17].

Next to the prespecified reactions, the reporter could choose an option to provide other AEFIs as free text. For all events, information was requested regarding latency time, seriousness according to international criteria, severity of the reaction according to a 5-point Likert scale (only on the consumer reporting form) [19], investigations, treatment, duration time and outcome. An event was considered to be serious if the patient died, the event led to (prolonged) hospitalisation, or a life-threatening event occurred. The patients were not asked if an event was ‘serious’, but only if one of the aforementioned conditions applied. Other information was asked about the batch number, previous COVID-19 vaccine(s), other suspected drugs, concomitant drugs, medical history, concomitant diseases, pregnancy, previous COVID-19 infections and patient characteristics such as date of birth.

Since vaccines are produced in batches, quality defects related to the production may have affected a specific batch. This would have necessitated the identification of specific batches. For this reason, when not known and with permission from the reporter, batch numbers were retrieved from the national vaccination registry (CIMS) [20] maintained by the National Institute for Public Health and the Environment (RIVM). In the reporting form, the citizen service number (BSN) is asked. This number is stored in a dedicated database and is not visible in the ICSR management system for privacy reasons. Once a week, a list of ICSRs with unknown batch numbers was sent to CIMS, and provided with batch numbers (if known in CIMS). This list was automatically imported to the specific ICSR in the ICSR management system and exported to Eudravigilance, the system for managing and analysing information on suspected adverse reactions to medicines that have been authorised or are being studied in clinical trials in the European Economic Area (EEA) [15]. If no batch number was found in CIMS, a second attempt was made. When the batch number retrieval was successful or failed after the second attempt, the citizen service number (BSN) was set to ‘privacy’ in the dedicated database.

With the linkage between adverse events and batch numbers of the vaccine administered, and the availability of data on background incidences of AEFI, we enabled the development of dedicated signal detection tools, including the analysis of batch-specific ICSRs [21].

A fully automatic process for ICSRs enabled the handling of the majority of common and known reported AEFIs that were prespecified on the reporting form. We decided to set up this new way of working to manage large numbers of straightforward ICSRs. The criteria for the automatic handling of ICSRs were as follows: the only suspected drug was a selected COVID-19 vaccination, concomitant drugs were coded automatically by selecting these from a list on the reporting form, AEFIs were reported as non-serious according to international criteria, and all selected AEFIs for automation had corresponding MedDRA LLT codes [17], which were mapped automatically on the pre-specified AEFIs in the reporting form. This automation process ran several times a day and made these ICSRs available for further processes such as signal detection within a day.

All other ICSRs were triaged daily by team PV assessors with expertise in vaccine AEFIs and signal detection (Fig. 2). A report was selected as high priority when it was marked as serious according to CIOMS criteria [22], concerned an Adverse Event of Special Interest (AESI) [23], or was deemed to have high signal value. This team (13.7 full-time equivalents (FTEs)) was also responsible for the signal detection and communication and included FTEs for coordination of tasks.

We arranged a ‘low-priority’ team (16.4 FTE) that handled ICSRs in a brief timespan on the most common AEFIs and reports we selected as ‘low-priority’ and a ‘high-priority team’, consisting of dedicated vaccines experts at Lareb, for assessing ICSRs that needed further clinical review and serious ICSRs. The total FTEs of this ‘COVID-19-team’ is 30.1 FTEs. For comparison, the vaccine team for the pharmacovigilance before COVID-19 consisted of 3.75 FTEs.

A possible safety signal in this context is defined as 'information about a new or known adverse reaction that may be caused by a COVID-19 vaccine and that warrants further investigation’ [24]. Signal detection generally relies on both the analysis of the clinical information and disproportionality analysis. In the latter approach a Reporting Odds Ratio (ROR) is calculated to see if the proportion of reported events for specified COVID-19 vaccines is compared to other drugs in the database. Although the disproportionality analysis offers the possibility to analyse large amounts of data, it is possible that results may be biased and may depend on the composition of the dataset. Interpretation should be carried out with caution and detailed knowledge of the underlying data should be taken into account. The influence of large amounts of data related to COVID-19 vaccines on disproportionality analysis was not known. To avoid any potential bias in the signal detection process, a first selection was based on the clinical content of reported cases. The ROR, stratified for vaccines, was one of the features in both weekly line-listings and the batch analysis, both described below.

The main driver of signal detection at our centre remains the clinical assessment of cases by the ‘high priority team’. On a daily basis, cases with ‘high signal values’, including AESIs and serious ICSRs, were discussed in a daily signal detection meeting. This team is supported by an external Clinical Advisory Board consisting of medical specialists in various fields of expertise, such as immunology, haematology, vascular medicine and internal medicine. In addition to the case-by-case analysis, a custom-made electronic reaction monitoring report was used to monitor all incoming ICSRs and to serve as an additional signal detection method in addition to case-by-case analysis. Available data elements in the electronic reaction monitoring report are provided in the Electronic Supplemental Material (ESM), Resource 1. Reporting rates per 100,000 per association for specific brands were incorporated into the report. When needed, further analyses of the AEFIs were conducted. If possible, background incidences of the condition in question were also included in the analyses. In the European project ACCESS [25], the background incidences of a large number of potential AEFIs have already been mapped out. For Observed/Expected analysis [26], stratification of background incidences by age groups and sex was performed. Data on the exposed population were provided by the RIVM based on the vaccination registry CIMS [20]. Background incidences specific to the Dutch population were provided on request by the PHARMO Institute. The PHARMO Database Network has detailed information of more than 4 million (≈ 25%) residents of a well-defined population in the Netherlands for an average of 10 years and makes it possible to use tailor-made and disease-specific cohorts [27]. Standardised Morbidity Rates (SMR) were used to compare the number of spontaneously reported cases of the association of interest (Observed) in the COVID-19 vaccinated population with the Expected number based on Dutch background incidence rates within a risk window following immunisation, stratified per vaccine, dose, sex and age.

SMR = O/E [28, 29]

E = (Nevents in PHARMO/Nperson years in PHARMO) * (risk period (days)/365) * Nvaccine exposure.

95% confidence intervals: \(\sqrt^)}\)/\(\sum E\); using Poisson distribution tables for low numbers of O (< 10)

The batch analysis compared the number of reports of one batch with the number of reports of the other batches of the same brand of vaccine. Batches with an LLROR > 1 (lower-level reporting odds ratio) were considered positive batches. The reports of these batches were investigated further to determine whether there were indications of a batch-related problem. The batch analysis was performed for fever and for injection-site infection. All positive batches were compared with the other batches of the same brand of vaccine in order to detect suspicious patterns. Among other things, the number of reports and the number of associated AEFIs, the number of serious reports and the number of serious AEFIs are examined. The reported serious PTs of the suspect batch were compared with the serious PTs of the other batches; this was also carried out with the non-serious PTs. The latency period and outcome were also considered. The age and gender distribution of the suspect batch were compared with the other batches of the same brand. Finally, a heat map was made of all reported PTs from all batches of a trademark and it was checked to see if there were noticeable patterns in the positive batches compared to the other batches.

For COVID-19 vaccine signal detection, Lareb worked in close collaboration with the Dutch Medicines Evaluation Board (CBG-MEB) and the National Institute for Public Health and the Environment (RIVM). These organizations provided input on signal detection activities and signal management. A weekly report was disseminated and discussed during a biweekly meeting with the MEB and RIVM. Safety signals were disseminated to MEB, who can take autonomous regulatory actions or forward the signal for further evaluation to the Pharmacovigilance Risk Assessment Committee (PRAC) or lead member states. The PRAC is the European Medicines Agency's (EMA) committee responsible for assessing and monitoring the safety of human medicines [30]. ICSRs on fatal outcomes were assessed within a 24-h timeframe and shared with the regulatory framework on a daily basis. Reports with a fatal outcome were each independently reassessed by two members of the Clinical Advisory Board. Potential safety signals were discussed within the regulatory framework to allow for regulatory action or additional studies.