We enjoyed reading the article “Comparison of the test-negative design and cohort design with explicit target trial emulation for evaluating COVID-19 vaccine effectiveness” by Li et al.,1 which sheds light on the differences between the target trial emulation approach2 and the test-negative design3 in estimating vaccine effectiveness (VE). As proponents and users of the target trials framework, familiar with its structured approach to the design of observational studies, and also having been involved in methods and applied research related to the test-negative design, we offer our perspective on certain bias-interpretability trade-offs that underlie both approaches to estimating VE, with implications for design and estimand choices.

According to the standard version of a test-negative design, researchers recruit or otherwise include data from individuals who are being tested for the infection of interest and whose infectious status is unknown at the time of recruitment.4,5 A key inclusion criterion is that all participants must have sought or received care for symptoms consistent with the vaccine-targeted infection (i.e., a case definition).6,7 This criterion may be either explicit or implicit, depending on the specific context of the study. Importantly, while several studies conducted during the COVID-19 pandemic deviated from this implementation of the test-negative design—either by including participants regardless of symptoms to estimate VE against infection8,9 or by excluding individuals with symptoms to estimate VE against asymptomatic infection10,11—we emphasize that such applications are not consistent with the validated design.6

As documented in the causal inference literature, a study design that conditions on postbaseline events, such as symptoms in the test-negative design, may be at risk of collider stratification bias.12 However, the possibility of this bias depends on specific aspects of the assumed underlying data-generating structure. In an ideal test-negative design setting, conditioning on symptoms may serve at least two critical roles: (1) it prevents collider bias by blocking the biasing pathway between infection status and vaccination that is opened by conditioning on receiving a test and/or care4 and (2) it restricts the study sample to individuals who exhibit preventive and other “healthcare-seeking” behaviors that could confound the association between infection and vaccination status.13 The latter is necessary, but not sufficient, to assume that the control group (i.e., those who test negative and are presumed to have another disease not targeted by the vaccine) accurately represents the general healthcare-seeking population from which we would have sampled controls in a standard case-control study, in terms of their exposure to vaccination and conditional on measured covariates, which is critical for the validity of the test-negative design.4

Leaving behind the idealized version of the test-negative design, bias can occur for several other reasons. For example, by sampling individuals with diverse disease severity14 and by conditioning on postvaccination covariates such as time-varying factors measured at the time of testing.1 However, the potential for bias may also change depending on what we consider to be the target parameter estimated by the test-negative design.

To illustrate this point, we refer the reader to the directed acyclic graph (DAG) shown in Figure 1, which represents a scenario in which the test-negative design recruits only individuals who are tested because of COVID-19-like symptoms [this is indicated by the boxes around the symptoms (W) and testing (T) nodes]. We allow for the vaccine under study to potentially affect the probability of being tested15–17 and the probability of presenting with COVID-19-like symptoms.18–20 We depict the possibility of unmeasured common causes of vaccination (V), infection [either the vaccine-targeted infection (I1) or some other infection (I2)], symptoms (W), and testing (T), such as health- and healthcare-seeking behaviors and COVID-19 susceptibility. According to this DAG, if we were to define SARS-CoV-2 infection (I2) as the outcome, our odds ratio estimate would be at risk of collider bias, for instance, due to conditioning on testing (T), a common effect of both symptoms (W) and vaccination (V). Conversely, if we define the outcome of interest as “being infected with SARS-CoV-2, experiencing COVID-19-like symptoms, and being tested for infection,” that is, Y = I2*W*T, the marginal risk ratio for this outcome, adjusted for measured covariates (C), is indeed identifiable.4 In fact, this is the commonly assumed target parameter of the test-negative design: if testing occurs in the context of receiving care, the outcome is usually described as “medically-attended and laboratory-confirmed (symptomatic) COVID-19.”21 Similarly, if testing occurs in an inpatient setting and hospitalization is due to symptoms, the outcome can be interpreted as “hospitalized for COVID-19.” These estimands have causal interpretations when there is no arrow from (U) to (V), no interference, positivity, and consistency.4

FIGURE 1.:

Directed acyclic graph representing the hypothetical relationship among baseline confounders (C), vaccination status (V), vaccine-targeted infection (I2), a test-negative condition such as another infection (I1), symptoms (W), and testing (T). The variable U represents all unmeasured common causes of vaccination (V), infection (I1 and I2), symptoms (W), and testing (T). We can condition the analyses on C, assuming it is measured accurately.

This brings us to an important point: the test-negative design cannot be convincingly used to estimate VE against infection (i.e., infections with or without associated symptoms). Even if we were to recruit people regardless of symptoms [i.e., removing the box around (W) in Figure 1] or estimate it only among those without symptoms, the test-negative design would still be conditioning on posttreatment and postoutcome events, and consequently, the results would still be at risk of collider stratification bias through multiple noncausal pathways linking vaccination with infection.22

We now consider some of the assumptions made by an observational cohort study in this context. The authors describe emulating a pragmatic target trial with a predetermined testing protocol, where individuals are required to be tested each time they present with symptoms of infection. Since asymptomatic infections would likely remain undetected, the analysis of a target trial with this testing protocol would estimate VE against (symptomatic) COVID-19. Because such a testing protocol may closely reflect real-world testing behavior, it could be credibly emulated in a cohort analysis, provided that we consider only symptomatic infections confirmed by a positive molecular or antigenic test result as the outcome of interest. On the other hand, if we conceive of a target trial with a routine testing protocol (i.e., individuals are tested regularly regardless of their symptoms) and assume that infections are equally likely to be detected in both treatment groups (e.g., vaccination does not reduce the probability of detecting an infection), then an analysis of this target trial could allow estimation of VE against infection. However, given that real-world data and testing protocols are predominantly symptom-driven, it is unlikely that such a testing strategy could be mapped to any routinely collected observational data, except in situations where routine testing is already in place and the rationale for testing is documented.

An additional consideration, particularly relevant for an infection outcome, is that observational data analyzed as a cohort may be subject to differential outcome misclassification. This may be a concern even with a perfect test for SARS-CoV-2 infection (i.e., assuming 100% sensitivity and specificity). In Figure 2, we present a DAG that illustrates this issue, where the measured outcome [measured infection (Y*)] is observed only after the test (T) and true infection (Y) have occurred, potentially leading to independent differential outcome misclassification along the V→T→Y* pathway and/or dependent differential outcome misclassification if common causes of vaccination (V) and testing (T) are present. Although measurement error in the outcome is often underestimated (because it is assumed to result only in a lack of precision23,24), this differential misclassification of the outcome may lead to bias in the VE estimate25,26 with magnitude and direction dependent on the structure and strength of the relationship between reasons for testing (e.g., general population vs. travelers or health care workers), vaccination, healthcare-seeking behavior, infection, and the presence of unknown factors.26 In contrast, the test-negative design incorporates (T) into the outcome, so that pathways V→T become part of the estimand rather than biasing pathways; however, common causes of (V) and (T) lead to confounding bias in the test-negative design. Incorporating symptoms into the outcome may lessen the possibility of strong causes of testing in the presence of symptoms and thus lessen confounding bias from arrows into (T).

FIGURE 2.:

Directed acyclic graph representing the potential for measurement error in the infection outcome ascertainment due to incomplete testing. The double-barred arrows indicate a deterministic relationship of the test outcome (Y*) given the receipt of test (T) and the presence of SARS-Cov-2 infection (Y), under the assumption of a perfect test for SARS-CoV-2. U represents unmeasured variables affecting T and V. Confounding variables of vaccination (V) and the infection outcome (Y) are not depicted. If U is not present, there is independent differential measurement error, leading to possible bias in the cohort design. If the U is present, there is dependent differential measurement error bias in the cohort and additional confounding bias in the test-negative design.

In conclusion, we believe that neither design, when applied to routinely collected observational data, can reliably estimate VE against infection. While the test-negative design controls for care-seeking behaviors, to some extent, by modifying the target population,21 its current modified implementations cannot adjust for time-dependent confounding of longitudinal treatment effects.1 Thus, test-negative design studies cannot differentiate between time-dependent effects, such as waning effectiveness due to time lapse since vaccination, and associations that change over time due to unadjusted time-dependent factors, such as new variants. In contrast, emulation of a target trial allows for time-dependent analysis of VE against symptomatic infection and more severe outcomes. However, a cohort design needs access to proxy information about care-seeking behavior to adequately control confounding. And indeed, because the test-negative design avoids bias by encapsulating both symptoms and testing into the outcome, a test-negative design may be preferable when the testing patterns in the real world are not reflective of those in the target trial. In general, both observational designs may struggle to adjust for confounders of vaccination and infection, since these include behavioral characteristics and even political affiliation typically not accounted for in either study. All caveats acknowledged, the implementation and interpretation of results from either design will depend on context, data structure and availability, and desired estimand. The practice of epidemiology and translation of the results into meaningful public health policies should consider the bias-interpretability trade-offs considered here.

ABOUT THE AUTHORS

MIREILLE SCHNITZER is an Associate Professor of Biostatistics at Université de Montréal. Her recent work on the test-negative design established nonparametric identifiability and semiparametric estimation.

EDGAR ORTIZ-BRIZUELA is a PhD student in Epidemiology at McGill University and an infectious disease physician by training. His doctoral project focuses on identifying potential sources of bias in test-negative design studies and developing strategies to mitigate them.

MABEL CARABALI is an Assistant Professor of Epidemiology at McGill University. Her research on infectious diseases involves methodological approaches to address measurement error.

DENIS TALBOT is a Professor of Biostatistics at Université Laval. He has been involved in monitoring COVID-19 vaccine effectiveness in Québec, Canada, during the pandemic.

REFERENCES 1. Li G, Gerlovin H, Figueroa Muñiz MJ, et al. Comparison of the test-negative design and cohort design with explicit target trial emulation for evaluating COVID-19 vaccine effectiveness. Epidemiology. 2024. 2. Hernan MA, Robins JM. Using big data to emulate a target trial when a randomized trial is not available. Am J Epidemiol. 2016;183:758–764. 3. Skowronski DM, Masaro C, Kwindt TL, et al. Estimating vaccine effectiveness against laboratory-confirmed influenza using a sentinel physician network: results from the 2005-2006 season of dual A and B vaccine mismatch in Canada. Vaccine. 2007;25:2842–2851. 4. Schnitzer ME. Estimands and estimation of COVID-19 vaccine effectiveness under the test-negative design: connections to causal inference. Epidemiology. 2022;33:325–333. 5. Jackson ML, Nelson JC. The test-negative design for estimating influenza vaccine effectiveness. Vaccine. 2013;31:2165–2168. 6. Sullivan SG, Khvorov A, Huang X, et al. The need for a clinical case definition in test-negative design studies estimating vaccine effectiveness. npj Vaccines. 2023;8:118. 7. Chua H, Feng S, Lewnard JA, et al. The use of test-negative controls to monitor vaccine effectiveness: a systematic review of methodology. Epidemiology. 2020;31:43–64. 8. Skowronski DM, Setayeshgar S, Zou M, et al. Comparative single-dose mRNA and ChAdOx1 vaccine effectiveness against severe acute respiratory syndrome coronavirus 2, including variants of concern: test-negative design, British Columbia, Canada. J Infect Dis. 2022;226:485–496. 9. Bruxvoort KJ, Sy LS, Qian L, et al. Effectiveness of mRNA-1273 against delta, mu, and other emerging variants of SARS-CoV-2: test negative case-control study. BMJ. 2021;375:e068848. 10. Chemaitelly H, Tang P, Hasan MR, et al. Waning of BNT162b2 vaccine protection against SARS-CoV-2 infection in Qatar. N Engl J Med. 2021;385:e83. 11. Tang P, Hasan MR, Chemaitelly H, et al. BNT162b2 and mRNA-1273 COVID-19 vaccine effectiveness against the SARS-CoV-2 Delta variant in Qatar. Nat Med. 2021;27:2136–2143. 12. Nguyen VT, Engleton M, Davison M, Ravaud P, Porcher R, Boutron I. Risk of bias in observational studies using routinely collected data of comparative effectiveness research: a meta-research study. BMC Med. 2021;19:279. 13. Sullivan SG, Tchetgen Tchetgen EJ, Cowling BJ. Theoretical basis of the test-negative study design for assessment of influenza vaccine effectiveness. Am J Epidemiol. 2016;184:345–353. 14. Ciocanea-Teodorescu I, Nason M, Sjolander A, Gabriel EE. Adjustment for disease severity in the test-negative study design. Am J Epidemiol. 2021;190:1882–1889. 15. Centers for Disease Control and Prevention. CDC updates travel guidance for fully vaccinated people. 2 April 2021. Available at: https://www.writediteach.com/images/Citing%20from%20a%20Digital%20Archive%20like%20the%20Internet%20Archive.pdf. Accessed 13 December 2023. 16. Occupational Safety and Health Administration. Employer Rights and Responsibilities Following a Federal OSHA Inspection. U.S. Department of Labor. n.d.. Available at: https://www.osha.gov/sites/default/files/publications/OSHA4159.pdf. Accessed 10 May 2023. 18. Antonelli M, Penfold RS, Canas LDS, et al. SARS-CoV-2 infection following booster vaccination: illness and symptom profile in a prospective, observational community-based case-control study. J Infect. 2023;87:506–515. 19. Antonelli M, Penfold RS, Merino J, et al. Risk factors and disease profile of post-vaccination SARS-CoV-2 infection in UK users of the COVID Symptom Study app: a prospective, community-based, nested, case-control study. Lancet Infect Dis. 2022;22:43–55. 20. Grana C, Ghosn L, Evrenoglou T, et al. Efficacy and safety of COVID-19 vaccines. Cochrane Database Syst Rev. 2022;2023:CD015477. 21. Dean NE, Hogan JW, Schnitzer ME. COVID-19 vaccine effectiveness and the test-negative design. N Engl J Med. 2021;385:1431–1433. 22. Ortiz-Brizuela E, Carabali M, Jiang C, Merckx J, Talbot D, Schnitzer ME. Potential biases in test-negative design studies of COVID-19 vaccine effectiveness arising from the inclusion of asymptomatic individuals. medRxiv. 2023. doi:10.1101/2023.11.16.23298633. 23. Keogh RH, Shaw PA, Gustafson P, et al. STRATOS guidance document on measurement error and misclassification of variables in observational epidemiology: Part 1-Basic theory and simple methods of adjustment. Stat Med. 2020;39:2197–2231. 24. Yi GY, Delaigle A, Gustafson P. Handbook of Measurement Error Models. CRC Press; 2021. 25. VanderWeele TJ, Hernan MA. Results on differential and dependent measurement error of the exposure and the outcome using signed directed acyclic graphs. Am J Epidemiol. 2012;175:1303–1310. 26. Valeri L. Measurement Error in Causal Inference. In: Handbook of Measurement Error Models. New York: CRC Press; 2021.