These case studies demonstrate that comparator group selection can influence observational study findings. Because precautions related to the use of these medications in patients with CV diseases or risk factors have likely resulted in a channeling bias, PDE5i users are not an appropriate comparator group when studying the association of TRT and MI risk, and triptan users are not an appropriate comparator group when evaluating adverse CV outcomes in migraine patients. The appropriateness of study design must always be considered in drug safety studies, and although statistical methods may alleviate some bias caused by flawed study design, they cannot be depended upon to eliminate the effects of confounding bias, especially when label information may be driving selective prescribing (e.g., patients with high CV risk or elderly high-risk patients channeled away from PDE5is or triptans due to labeling restrictions).

Comparison of TRT-treated versus untreated men in our previous publication revealed no association between TRT use and AMI in patients > 65 years old or for patients with prior CVD (Table 2) [22]. The contrasting findings in the present TRT case study demonstrate the imposition of potential channeling bias as a result of using PDE5i-treated patients as a comparator group; although results using Cox regression analysis showed no difference in MI risk between TRT-treated and PDE5i-treated patients, a significantly increased risk of MI was erroneously observed among TRT-treated patients > 65 years and TRT-treated patients with baseline CVD likely due to channeling bias. In order to address confounding by severity, a treated active comparator with the same underlying disease is better than an untreated or inactive comparator. It is also important to consider a drug/drug class at the same stage of disease severity and with the same or similar contraindications. These are all important considerations when designing a real-world evidence study.

The TRT case study reinforces methodological concerns; for example, that channeling bias may have existed in previously published research. Selecting PDE5i users as a comparator group likely introduced channeling bias into the study by Finkle et al. [2], which found TRT use to be associated with an overall two-fold increased risk of MI compared with PDE5i use among TRT-treated elderly patients and reported an approximately three-fold increase in incidence rate among TRT-treated younger men with pre-existing CV conditions [2]. Prescribers are aware that PDE5is are contraindicated in patients taking organic nitrates and should not be prescribed to men for whom sexual activity is inadvisable due to their CV status [7,8,9]; similar information was not provided to prescribers when considering treatment with TRT. Considering results of a meta-analysis suggesting low endogenous testosterone increased CV risk and risk of CVD death [23], it is reasonably possible that patients with worse CV conditions were treated with TRT. Because TRT has been associated with improved sexual function among men with erectile dysfunction and low testosterone [24, 25], it is also possible that healthcare practitioners prescribe TRT to hypogonadal patients with high CVD risk and erectile dysfunction symptoms and prescribe PDE5is to patients without known heart conditions. This was evident in the TRT case study, wherein baseline characteristics of pre-matched TRT-treated patients suggested worse health (i.e., significantly increased hyperlipidemia) compared with PDE5i-treated patients. Differences in study designs, patient populations, comparator groups, availability of laboratory tests, and statistical analyses may explain the inconsistencies in study findings.

Migraine has been associated with increased CV risk including increased risk of ischemic stroke, MI, angina or coronary revascularization, and CV mortality [26,27,28]. This retrospective cohort study of acute treatment for migraine revealed a lack of association between triptan use and CV risk. Although multiple comparators were chosen, potential channeling bias resulting from contraindications stated in the triptan product labels may have been strong enough to select the heathiest patients from among all migraine patients on acute medications. These findings are in line with those of a previous analysis in which we used a prediction model to assess baseline CV risk among migraine patients prescribed triptans in comparison with those prescribed opiates or NSAIDs, as well as untreated migraine patients and general non-migraine patients [19]. In that analysis, patients who were prescribed triptans had lower CV risk than those prescribed opiates or NSAIDs (in fact, they were the healthiest among all treatment groups) [19]. We conclude it is likely that patients with migraine who also have high CV risk are being prescribed triptans at lower rates than they are prescribed opiate or NSAID alternatives.

Similar to the findings of the TRT case study regarding patients with low testosterone, the findings of the retrospective triptan cohort study support the hypothesis that channeling bias may occur in choosing whether or not to prescribe triptans for patients with migraine. Given the CV warnings included in current triptan labeling, the findings of so-called protective effects of triptans for MI and stroke compared with alternative treatments suggest channeling of migraine patients at high CV risk away from triptans in favor of alternative therapies.

Each of these observational studies is susceptible to the potential for confounding bias. In real clinical practice, prescriptions are not administered at random, and although the potential for confounding bias can be adjusted through careful selection or using statistical methods, bias might remain because no method can alleviate all potential for selection bias in a study. For example, as a risk factor for many comorbidities, diabetes severity can cause residual confounding or selection bias in real-world comparative studies of anti-diabetes medication. In a cohort study evaluating non-fatal major adverse cardiovascular events (MACE), adjustment of baseline characteristics did not remove baseline differences between metformin and insulin, as illustrated by increased HR for pre-exposure MACE (HR 1.85) [29]. Similarly, in the TRT/hypogonadism case study, confounding by indication/severity could have been introduced if the PDE5i or untreated cohort had higher serum testosterone levels versus the treated cohort, because lower testosterone levels have been linked to higher rates of CV events [30,31,32] and those getting treatment likely have lower levels than untreated patients. In the triptan/migraine cohort study, multiple active comparators, new user design, propensity score techniques, and sensitivity analysis were used in an effort to minimize confounding bias. Typically, having multiple comparators can provide different population characteristics and is a good sensitivity analysis to assess for potential confounding (as with the PDE5i example, which showed no increased risk in untreated patients and an increased risk with PDE5i use). However, this was not the case for triptans because healthier patients were selected for prescription of triptans because of the potential vasoconstrictive effect of triptans. The fact that they had fewer comorbidities, CV risk factors, and hospitalizations and emergency room visits compared with the prescription NSAID, opiate, or untreated cohorts might have contributed to the false impression of a triptan protective effect. In addition, both prescription NSAIDs and opiates have demonstrated adverse effects on CVD previously [11, 33]; hence, all comparisons showed a false protective effect.

Both studies were informed by several database limitations. First, it is difficult to verify the validity of diagnosis codes and to refine statistical analyses owing to the limited clinical details in the MarketScan database. Second, data regarding important confounding variables (e.g., smoking, alcohol use, body weight, body height, and social economic status) are not available in the claims database. Such variables are likely to be equally distributed among the study cohorts; in addition, propensity score matching was used to reduce any differences in distribution.

Measurement bias is a third important limitation. Because claims data are generally collected for the purpose of payment rather than research, the presence or absence of disease may not be accurate; similarly, diagnostic codes may be incorrectly coded or included as rule-out criteria rather than actual disease. Furthermore, diagnoses, medical procedures, and medicine dispensing were only included in the database if corresponding billing codes were generated.

Fourth, MarketScan claims databases are based on a large convenience sample, rather than a random sample. The included data generally originate from large employers, and medium and small firms are not well represented. Because the sample is not random, it may contain biases and may not be highly generalizable to other populations. Further, migraine diagnoses are not all recorded in claim databases, including MarketScan. According to Kolodner et al., medical and pharmacy claims were highly specific and moderately sensitive when used to identify patients with migraine [33]. This may lead to misclassification of exposure status in the comparison with non-migraine patients and likely draw the association towards a null finding.

Lastly, regarding the retrospective cohort study on acute treatment for migraine, only prescribed medications are recorded in the MarketScan database; it does not contain any over-the-counter NSAID medications, which can be used widely to treat migraine, pain or inflammation. The lack of over-the-counter drug information may result in unmeasured confounding, which could potentially impact the study findings. Furthermore, due to lack of information on over-the-counter NSAIDs, there may also be concerns about misclassification with respect to exposure status and exposure window. It should also be noted that it is difficult to estimate the exposure window for medications prescribed on an as-needed basis. Claims data do not include information about which exact days medication was consumed, whether it was taken as prescribed, or whether it was consumed at all. To avoid potential misclassifications when assessing drug exposure, exposure windows were defined using an ‘as-treated’ approach based on prescription fills, and 30-day, 60-day, 90-day, and ITT analyses were assessed. To further confirm the above negative findings, a positive control analysis (untreated migraine versus general non-migraine patients) was constructed in which some statistically significant increased risk for stroke was observed.