2.1 Data Source

For this study, we used the US Food and Drug Administration (FDA) Adverse Event Reporting System (FAERS) database [10]. The FAERS contains reports of suspected adverse drug reactions based on voluntary submissions by healthcare providers and patients and mandatory submissions by pharmaceutical companies. Information on these reports is publicly available in seven different data files linkable via a unique identifying number: (i) a demographic file with information about the patient, including age, sex, and weight; (ii) a drug file with the role of the drug in the adverse event (primary suspect drug, secondary suspect drug, concomitant drug, or interacting drug), the active ingredient and/or brand name(s); (iii) a reaction file with clinical information about the adverse event (coded via ‘preferred terms’ from the Medical Dictionary for Regulatory Activities [11]); (iv) an outcome file with information about the severity of the adverse event and potential complications; (v) a report source file; (vi) a therapy dates file with start and end dates for the reported drugs; and (vii) an indications for use file with information about the underlying indications for the reported drugs (coded using preferred terms). The design and the analyses described below were based on a protocol that was finalized prior to the initiation of the study.

2.2 Rationale for the Choice of DECs

The two DECs used for the purposes of this study were (i) the direct oral anticoagulant (DOAC) rivaroxaban and hepatotoxicity and (ii) the sodium-glucose cotransporter 2 (SGLT2) inhibitor canagliflozin and acute kidney injury (AKI). These DECs were chosen because of their high propensity for bias, specifically for false-positive signals because of the potential for different types of stimulated reporting.

2.2.1 Rivaroxaban and Hepatotoxicity

Rivaroxaban was one of the first two DOACs to be approved for the prevention of ischemic stroke among patients with non-valvular atrial fibrillation (the other one was dabigatran). Because ximelagatran, the first DOAC to be studied in a phase III trial, was not approved (or was removed from the market shortly after approval) because of its hepatotoxic risk [12], there were initial concerns with respect to the hepatic safety of DOACs.

The Randomized Evaluation of Long-Term Anticoagulation Therapy (RE-LY) and the Rivaroxaban Once Daily Oral Direct Factor Xa Inhibition Compared with Vitamin K Antagonism for Prevention of Stroke and Embolism Trial in Atrial Fibrillation (ROCKET AF) randomized controlled trials did not yield any hepatotoxic signals for dabigatran or rivaroxaban [13, 14]. Moreover, several subsequent pharmacoepidemiologic studies corroborated the absence of a hepatotoxic risk for dabigatran and rivaroxaban when compared with warfarin, a vitamin K antagonist (VKA) [15,16,17]. However, an early pharmacovigilance study by Raschi and colleagues showed signals for liver injury and liver failure for rivaroxaban (not for dabigatran) [18]. Therefore, we hypothesized these signals to be false positives resulting from notoriety bias, a bias known to occur when safety concerns such as those mentioned above exist.

2.2.2 Canagliflozin and AKI

Canagliflozin was approved in 2013 as the first SGLT2 inhibitor, a class of oral antidiabetic drugs used in the management of type 2 diabetes mellitus. Canagliflozin was followed by dapagliflozin and empagliflozin in 2014. By interfering with the co-uptake of glucose and sodium in the proximal nephron, SGLT2 inhibitors increase sodium delivery to the distal nephron, resulting in afferent arteriole vasoconstriction and a decreased glomerular filtration rate. Thus, potential nephrotoxicity of these medications was an initial safety concern.

Indeed, in June 2016, there was a safety warning issued by the FDA on the use of SGLT2 inhibitors and the risk of AKI [19]. The safety warning mentioned that between March 2013 and October 2015, FAERS documented 101 cases of AKI with a temporal relationship mostly with canagliflozin (73 patients) but also with dapagliflozin (28 patients). Despite these concerns, randomized controlled trials on the efficacy and safety of SGLT2 inhibitors in patients with chronic kidney disease (with or without type 2 diabetes) observed a lower risk of kidney failure and other serious renal events when compared with placebo [20,21,22]. Moreover, several pharmacoepidemiologic studies did not show an increased risk of AKI or serious adverse events pertaining to the urogenital system in general associated with the use of SGLT2 inhibitors, when compared with the use of dipeptidyl-peptidase-4 (DPP-4) inhibitors [23,24,25]. Despite the probable absence of an excess nephrotoxic risk with SGLT2 inhibitors overall and with canagliflozin in particular, we hypothesized that the initial safety concerns and the publication of the regulatory safety warning could lead to stimulated reporting and false-positive signals.

2.3 Study Period

For the rivaroxaban and hepatotoxicity DEC, we extracted and linked the quarterly raw data submitted to FAERS between 1 July, 2011 (the date of rivaroxaban approval by the FDA) and 30 June, 2023. For the canagliflozin and AKI DEC, the same process was repeated for the time period between 1 April, 2013 (the date of canagliflozin approval by the FDA) and 30 June, 2023. We utilized the most recent updates for each report assigned to the quarter in which the report was first made available by the FDA.

Rigorous data cleaning included the correction of misspelled drug names, the removal of duplicate reports identified by merging active ingredient and trade medication names, and the flagging of reports with the same reporting country, sex, event date, age, adverse events, and drugs. We only included the most recently updated data at the time of the first report and also removed reports with insufficient or inconsistent information. Finally, we considered only reports where the study drug was listed as a primary suspect drug or secondary suspect drug.

2.4 Outcome Definition

We aimed to maximize the specificity of our outcome definitions. Hence, we used preferred terms explicitly referring to the adverse events of interest. For the rivaroxaban and hepatoxicity DEC, we defined hepatotoxicity using the preferred terms “acute yellow liver atrophy,” “allergic hepatitis,” “alloimmune hepatitis,” “autoimmune hepatitis,” “cholestatic liver injury,” “chronic hepatitis,” “drug-induced liver injury,” “hepatic cytolysis,” “hepatic infiltration eosinophilic,” “hepatic necrosis,” “hepatic steatofibrosis,” “hepatic steatosis,” “hepatitis,” “hepatitis acute,” “hepatitis chronic active,” “hepatitis chronic persistent,” “hepatitis fulminant,” “hepatitis toxic,” “hepatocellular injury,” “hepatotoxicity,” “immune-mediated hepatic disorder,” “immune-mediated hepatitis,” “liver injury,” “mixed liver injury,” “non-alcoholic steatohepatitis,” “non-alcoholic fatty liver disease,” and “steatohepatitis”. To emulate the study by Raschi and colleagues [18], we also assessed liver failure, a severe form of hepatotoxicity, which was defined using the preferred terms “acquired hepatocerebral degeneration,” “acute hepatic failure,” “acute on chronic liver failure,” “chronic hepatic failure,” “coma hepatic,” “hepatic encephalopathy,” “hepatic failure,” “hepatogenous diabetes,” “hepatorenal failure,” “hepatorenal syndrome,” and “subacute hepatic failure”. For the canagliflozin and AKI DEC, we defined AKI using the preferred term “acute kidney injury”.

2.5 Comparators

We considered the influence of three different reference sets or comparators. The first comparator was the ‘full-data reference set’ that contained all other drugs in FAERS except for the study drug with no restrictions. This comparator acted as our benchmark for relative comparison against the two comparators that implemented restrictions. The second comparator was the ‘active-comparator reference set’ that included only reports with at least one drug with the same indication as the study drug. Hence, this comparator included both the drugs from the active comparator class (VKAs in the case of rivaroxaban and DPP-4 inhibitors in the case of canagliflozin), but also all other drugs from the same class as the study drug (non-rivaroxaban DOACs in the case of rivaroxaban and non-canagliflozin SGLT2 inhibitors in the case of canagliflozin). Reports that contained both the study drug and another drug with the same indication were excluded from the analysis. Finally, the third comparator was the ‘active-comparator class-exclusion reference set,’ a combination set that included only drugs from the active comparator class, while removing all other drugs from the same class as the study drug. Here, rivaroxaban was compared to VKAs and canagliflozin was compared to DPP-4 inhibitors. Reports containing both the study drug and a drug from the class of active comparators were excluded. An overview of the different comparators can be found in Table 1.

Table 1 Description of the three different comparators for the two drug-event combinations

For both DECs, we hypothesized that the use of restricted comparators would decrease confounding by indication but increase the potential for reporting biases. For the rivaroxaban and hepatotoxicity DEC, we expected the impact of confounding by indication to be minor given the unclear association between thrombosis (the main indication for DOACs) and hepatotoxicity. We also expected the impact of reporting biases such as notoriety bias to be major, especially in the early years given the initial safety concerns. For the canagliflozin and AKI DEC, we expected the impact of confounding by indication to be strong given the well-established association between type 2 diabetes (the main indication for SGLT2 inhibitors) and kidney disease. We also expected the impact of reporting biases such as notoriety bias to be moderate and to vary over time depending on the timepoint of the FDA safety warning.

2.6 Calendar Time Restrictions

We also restricted the analysis time periods to consider meaningful events that may have influenced reporting behavior, allowing a further elucidation of the impact of different biases over time. For both DECs, the follow-up in the primary analyses ended 30 June, 2023, which was the latest date of data availability. For the rivaroxaban and hepatotoxicity DEC, we restricted the follow-up in a sensitivity analysis to September 2013, thereby mimicking the pharmacovigilance study by Raschi and colleagues that detected safety signals for rivaroxaban [18]. The rationale was that newly marketed drugs should have a boost in reporting shortly after market entry [26]. This early boost could be even more pronounced with rivaroxaban given the initial concerns over potential hepatotoxicity with DOACs. For the canagliflozin and AKI DEC, we restricted the follow-up in a sensitivity analysis to June 2016, that is until the publication of the safety warning by the FDA. The rationale was that considering only the time until the safety warning should minimize stimulated reporting.

2.7 Disproportionality Analysis Methods

To empirically study the influence of comparator selection on signal detection, we derived estimates of disproportionate reporting for each DEC with each of the comparators and within each calendar time interval, as defined above. These estimates were computed with commonly used disproportionality analysis methods including the proportional reporting ratio (PRR), the reporting odds ratio (ROR), and the relative reporting rate, the target parameter for the Bayesian approaches known as the Bayesian Confidence Propagation Neural Network (BCPNN) and the Multi-item Gamma Poisson Shrinker (MGPS) [27].

The PRR is a reporting-based analog of the relative risk, which compares reporting rates of a specific adverse event in those who reported the drug of interest with the drugs in the chosen reference set. The ROR is a reporting-based analog of the odds ratio, which compares the odds of reporting of an adverse event with the drug of interest to the odds of the same adverse event with the drugs in the reference set. The relative reporting rate compares the rate of reporting the ADR of interest against what would be expected under the assumption of independent reporting.

Both the PRR and the ROR tend to produce false-positive signals in settings of sparse data. In such settings, Bayesian implementations of the relative reporting rate such as BCPNN and MGPS can shrink the estimates because they allow for the specification of a priori knowledge, or beliefs, about a DEC. This then enables the calculation of estimates with reasonable properties even when only small amounts of data are available [28]. Of note, as more data are collected, these estimates become less dependent on the prior information and more reflective of the information housed in the data; as a result, when sample sizes become large enough, they will resemble their frequentist counterparts. Hence, Bayesian estimators possess favorable properties in the context of detecting drug safety signals for rare events, or early on in the marketed period of a new medication.

A criticism of these estimators is the potential for specifying prior subjective beliefs that may make a drug appear more or less safe. However, in our analyses, and in pharmacovigilance in general, we used the BCPNN that specifies priors based on conservative assumptions about the reporting relationship [29]. The method incorporates an independence assumption that mitigates concerns surrounding false-positive signals [29]. Of note, both DECs used in our analyses were not expected to produce small enough counts to produce major discrepancies in the signals detected between the PRR, ROR, and BCPNN algorithms.

The MGPS is empirical Bayes, that is, the a priori information is not provided by the investigator as with the BCPNN but is estimated directly from the data. The portion of the MGPS algorithm that attempts to introduce the prior information is conducted on the analytic dataset, which makes convergence of optimization routines unlikely in small subsets of a spontaneous reporting database; such as in the case of restricted reference sets. Hence, we excluded this algorithm from our analyses given that our purpose was to ascertain the influence of reference set selection on relative reporting bias. Indeed, the potentially limited performance of the numerical procedure would make an evaluation of the impact of bias on the disproportionality analysis estimates with the MGPS challenging.

2.8 Quasi-QBA Analysis

We quantified the absolute bias comparing PRR, ROR, and IC between each of the two restricted comparators and the unrestricted comparator, which acted as a benchmark, for both DECs. To this end, we converted PRR and ROR to a logarithmic scale to enhance comparability with IC, which is already logarithmically converted. Then, we used the formula DPRES−DPUNRES, where DPRES is the disproportionality analysis estimate in the restricted comparators and DPUNRES is the disproportionality analysis estimate in the unrestricted comparator. We also quantified the absolute bias between the calendar time-restricted estimates.

In general, in order to make a generalizable recommendation for the selection of a comparator, we would expect to see similar relative relationships in the directionality of the estimates between the benchmark analysis and each of the restricted comparators for both DECs. Of note, for the two calendar time intervals, we expected to see stimulated reporting immediately after the noted external events and described the impact of each comparator on this bias by comparing between the time intervals for each DEC. All analyses were conducted using R (R Foundation, Vienna, Austria).