Identification of eligible studies

Of the 3113 records that were identified and screened for relevance at the title and abstract stage, three were excluded as duplicates and 1338 were excluded on an abstract screen (Fig. 1). Then, 1963 full-text articles were screened for inclusion against the eligibility criteria, and 308 studies met the criteria required for the overall review sample (see flow chart for details of reasons why studies were excluded, mostly (n = 191) due to not reporting a relative risk or risk difference). These 308 studies were RCTs that reported either an unadjusted or covariate-adjusted relative risk, risk difference or both. Of these 308 studies, 131 were identified to report either a covariate-adjusted relative risk, risk difference, or both and were eligible for inclusion in the nested review sample. None of the included studies reported on more than one RCT in each trial report. 

Fig. 1

PRISMA flow diagram of the selection process for eligible studies. The overall review sample includes RCTs published between January 1, 2018, and March 11, 2023, in selected high-impact journals, with a primary binary outcome, that report either a relative risk (RR) or risk difference (RD) (unadjusted or adjusted) as a summary measure. The numbers reported for unadjusted RR and RD are not mutually exclusive to the reporting of adjusted RR and RD. RR, relative risk; RD, risk difference. The nested review sample is a sub-sample of RCTs identified from the overall review sample that reports a covariate-adjusted RR or RD for the primary, secondary, or exploratory analysis. The numbers reported for adjusted RR and RD are not mutually exclusive

Agreement across data extraction

The percentage agreement across the two independent data extractions for the 21 studies that were extracted in duplicate was 92.8%, and unweighted Kappa was 0.9 (95% CI: 0.80–0.99). Most discrepancies were due to a lack of clarity in the description provided by the authors regarding the methods used to derive summary measures for the unadjusted analysis (Table A4). In many cases, it was unclear how the confidence intervals were derived. Other discrepancies were due to a lack of clarity in approach when both the relative risk and risk difference were reported. It was also commonly unclear whether the p-value reported was for the relative risk or risk difference.

Characteristics of included studies

Of the 308 RCTs in the overall review sample, most trials were published in JAMA (108/308, 35%) or NEJM (108/308, 35%) compared to Lancet (75/308, 24%) or BMJ (17/308, 6%) (Table 1). The studies were all published between 2018 and 2023. Most (293/308, 95%) were multi-centre studies with an average sample size in the control and intervention arms of 357 (IQR: 158–717) and 367 (IQR: 176–729), respectively. Using a restricted method of randomisation was common (213/308, 69%), with the average number of covariates used in the randomisation reported to be 2 (IQR: 1–3). Of those that used restricted randomisation (n = 213), about half (107, 50%; 95% CI: 43–57%) reported adjusting for all those covariates used in the restriction; whilst an additional 18 (9%; 95% CI: 5%, 13%) partially adjusted for some of the covariates; in about a fifth (45, 21%; 95% CI: 16–27%) the analyses did not adjust for these covariates and in the remainder (43, 20%; 95% 15–26%), it was unclear. Of the 308 RCTs, around half (150/308, 49%; 95% CI: 43–54%) reported a covariate-adjusted relative risk or risk difference.

Table 1 Characteristics of studies included in the overall review sampleMethods for estimating unadjusted relative risks and risk differencesUnadjusted relative risks

Of the 308 studies, 95 reported an unadjusted relative risk (Table 2). In the majority (n = 65, 68%; 95% CI: 58–77%) of these reports, the method used to estimate the confidence intervals for the unadjusted relative risk was unclear. Of the 30 RCTs that reported an unadjusted relative risk and where the reporting of the method for the confidence interval was clear, the most common method used was the log-binomial model, used in 21 (70%; 95% CI: 50–85%) of the studies; followed by modified Poisson, used in 7 (23%; 95% CI: 11–43%) of the studies; and only in one (3%; 95% CI: 0–19%) study was marginal standardisation reported to be used. See Table A5 for a granular description of the methods identified. Of the 95 studies that reported an unadjusted relative risk, almost all studies reported point estimates and upper and lower confidence intervals. However, whilst p-values for the unadjusted relative risk were reported for most studies (77, 81%), they were not always reported. No studies reported standard errors of effects.

Unadjusted risk differences

Of the 308 studies, 194 RCTs reported an unadjusted risk difference. In the majority (n = 139, 72%; 95% CI: 65–78%) of these 194 reports, the method used to estimate the confidence intervals for the unadjusted risk difference was unclear (Table 2). Of the 55 RCTs that reported an unadjusted risk difference and where the reporting of the confidence interval was clear we did not identify a universally more common approach: the binomial model was used in 9 (16%; 95% CI: 8–29%) studies; the linear model used in 6 (11%; 95% CI: 4.5–23%) studies; marginal standardisation used in 4 (7%; 95% CI: 2–18%) studies; and modified Poisson used in 2 (4%; 95% CI: 1–14%) studies. Moreover, we identified that many other, different, approaches were used as in 34 (62%; 95% CI: 48–74%) of the studies, the approach was classified as ‘other’. See Table A6 for a granular description of the methods identified. Again, of the 194 RCTs that reported an unadjusted risk difference, most reported point estimates and upper and lower confidence intervals. However, only around half of the studies, 127/194 (65%), reported p-values for the unadjusted risk difference; and no studies reported standard errors of estimated effects.

Table 2 Methods used for estimating unadjusted and adjusted relative risks and risk differences and completeness of reportingMethods used to estimate the covariate-adjusted treatment effectsAdjusted relative risks

Of the 308 studies, 82 reported an adjusted relative risk (Table 3). In about a quarter of these reports, the method used was unclear, 17 (21%; 95% CI: 13–31%). When the reporting was clear, the log-binomial model was the most common method used in 42 (65%; 95% CI: 52–76%) studies; then modified Poisson which was used in 19 (29%; 95% CI: 19–42%); and marginal standardisation used in two (3%; 95% CI: < 1–12%) studies. See Table A5 for a granular description of the methods identified. Most studies (82, 100%) reported point estimates, and upper and lower confidence intervals (82, 99%). However, p-values for the adjusted relative risk were reported less frequently (reported in 66 (80%) studies); and no studies reported standard errors.

Adjusted risk differences

Of the 308 studies, 92 reported an adjusted risk difference. The method used was unclear in a considerable proportion, 36 (39%; 95% CI: 29–50%). When the reporting was clear, the binomial model, used in 27 (48%; 95% CI: 35–62%) studies, was the most common method. Then marginal standardisation was used in 12 (21%; 95% CI: 12–35%) studies; then linear model which was used in 6 (11%; 95% CI: 4–23%) studies; and modified Poisson used in 4 (7%; 95% CI: 2–18%) studies. However, a small proportion of methods in 7 (13%; 95% CI: 6–25%) were classified as other. Most studies (91, 99%) reported point estimates, upper confidence intervals (88, 96%) and lower confidence intervals (87, 95%). However, p-values for the adjusted risk difference were reported less frequently (reported in 53 (58%) studies); and no studies reported standard errors. See Table A6 for a granular description of the methods identified.

Whether covariate-adjusted effects are reported for primary analyses

Of the studies included in the nested review sample, 82 studies reported a covariate-adjusted relative risk, and 92 reported a covariate-adjusted risk difference (Table 3). Of the 82 studies that reported an adjusted relative risk, in 67 (82%; 95% CI: 71–89%) studies, the adjusted relative risk was reported as the primary analysis; in 5 (6%; 95% CI: 2–14%) studies this was for a secondary analysis; and in 10 (12%; 95% CI: 6–22%) studies this was for an exploratory or sensitivity analysis. Similarly, of the 92 studies that reported a covariate-adjusted risk difference, in 74 (80%; 95% CI: 71–88%) studies this was for the primary analysis; in 6 (7%; 95% CI: 3–14%) studies this was for a secondary analysis; and in 12 (12%; 95% CI: 7–22%) studies this was for an exploratory or sensitivity analysis.

Table 3 Current practice related to covariate-adjusted analysis in the nested review sampleJustification of the choice of covariates used for adjustment

For the 82 studies that reported an adjusted relative risk, in the vast majority (73, 89%; 95% CI: 80–95%) of these 82 studies, the covariates used in the adjustment were pre-specified (which included randomisation covariates). In only a few (4, 5%; 95% CI: 2–13%) studies, covariate adjustment resulted from data-driven post-hoc analyses or unclear approaches (5, 6%; 95% CI: 2–14%). In addition, 34 (41%; 95% CI: 31–53%) studies reported adjusting for covariates other than those included in a restricted randomisation. The average number of additional covariates (i.e. in addition to those adjusted because they were used in the randomisation) used in these studies was 3 (IQR: 2–5).

Of the 92 studies that reported an adjusted risk difference, a restricted method of randomisation was used in 85 (92%; 95% CI: 84–97%) studies. For these studies, the reported rationale for choosing these covariates was mostly pre-specification in 84 (91%; 95% CI: 83–96%); rather than data-driven in 2 (2%; 95% CI: < 1–8.4%) or unclear approaches in 6 (7%; (95% CI: 3–14%). In addition, 33 (36%; 95% CI: 26–47%) studies reported adjusting for covariates other than those included in a restricted randomisation. The average number of additional covariates used in these studies was 3 (IQR: 2–4).

Comparison of adjusted and unadjusted effectsRelative risks

Of the 82 studies that reported covariate-adjusted relative risk, 41 reported both an unadjusted and adjusted relative risk (Fig. 2). The adjusted and unadjusted point estimates were mostly observed to be similar for each study, with occasional larger differences that went in both directions (sometimes the adjusted relative risk was larger than the unadjusted relative risk and vice versa). A similar pattern was observed for standard errors and p-values: mostly these were similar between the adjusted and unadjusted analyses, but sometimes there were larger differences, and these differences went in both directions—so that sometimes the standard error from an adjusted analysis could be smaller than from the unadjusted analysis and sometimes larger.

Fig. 2

Comparison of adjusted to unadjusted relative risks. Footnotes: relative risk; CIs, confidence intervals. If the point falls below the line of equality, then the value is smaller under the adjusted analysis than under the unadjusted analysis (as anticipated per theory). However, if the point is above the line of equality, then the value is larger under the adjusted analysis than under the unadjusted analysis

Risk differences

Of the 92 studies that reported covariate-adjusted risk differences, 42 reported both an unadjusted and adjusted risk difference. The reader is referred to in Fig. 3, and the findings are similar to those of the relative risk.

Fig. 3

Comparison of adjusted to unadjusted risk differences. Footnotes: RD, risk difference; CIs, confidence intervals. If the point falls below the line of equality, then the value is smaller under the adjusted analysis than under the unadjusted analysis (as anticipated per theory). However, if the point is above the line of equality, then the value is larger under the adjusted analysis than under the unadjusted analysis