Broad assessment of risk of biasOverall, all but three of the trials were assessed as at high risk of bias and only one was assessed at low risk of bias (Table 3, Figure 1). Most trials were assessed as high risk on one (9, 22.5%) or two (14, 35%) domains; with a smaller number being assessed as risk on up to 4 (6, 15%) or 5 (2.5%) domains. Breaking down these assessments into finer categories (Supplementary Tables 1a to 5) helps identify the design features associated with these risks of bias. We next consider each domain separately.

Table 3Risk of bias assessment by broad domains of risk

Figure 1Percentage of papers in each risk category across the broad domains of risk

Domain 1a bias arising from the randomisation process: Around half of the trials (21, 52.5%) were assessed as being at high risk of bias due to the randomisation process. Whilst all were assessed to use a random method to allocate clusters to treatment conditions, many (21, 52.5%) were assessed as not having concealed the allocations (i.e., not clearly reporting randomisation by someone independent, or using a trials unit, or not using some acceptable concealment mechanism, such as internet-based randomisation or sealed envelopes). Most trials (30, 75%) did not report any cluster-level characteristics to allow any assessment of balance of the randomisation process.

Domain 1b bias arising from identification or recruitment of participants within clusters: A large majority of the trials (27, 67.5%) were assessed as at risk of bias due to the timing of identification and recruitment of participants. Most trials (36, 90%) were assessed as identifying or recruiting participants after randomisation and most (27, 67.5%) were assessed to include participants in such a way that selection could have been affected by knowledge of the intervention. As shown in Supplementary Table 6, this is because many trials both recruited participants post randomisation and those recruiting participants were not reported to be blind to the intervention. In some trials (15, 37%), we identified baseline imbalances that suggest differential identification or recruitment of individual participants between arms.

Domain 2 bias due to deviations from intended interventions: Most trials (34, 85%) were at low risk of bias due to deviations from the intended interventions. However, in a large number of trials, we deemed that participants were aware that they were in a trial (27, 67.5%) and aware of their assigned intervention (20, 50%), as did trial personnel (34, 85%). Despite this, only a minority of trials (8, 20%) were assessed as showing evidence of deviations from the intended intervention beyond what would be expected in usual practice; and in only a few trials (6, 15%) were these deviations from intended intervention unbalanced between groups and assessed as likely to have affected the outcome (Supplementary Table 7). Here we assumed that a deviation of the intended intervention occurred if more than 10% of the participants were reported not to have received the intended intervention condition. In all trials, most clusters and participants were reported to be analysed according to randomisation (i.e., by intention to treat).

Domain 3 bias due to missing outcome data: Most trials (33, 82.5%) were assessed as at low risk of bias due to missing outcome data, mostly because missing data arose infrequently: only in a small number of trials (9, 22.5%) was the outcome data unavailable for more than 10% of participants. In a small number of cases (4, 10%) outcome data were deemed to be differential across treatment arms.

Domain 4 bias in measurement of the outcome: Most of the trials were assessed as being at low risk of bias due to measurement of the outcome (31, 77.5%), although some (9, 22.5%) were assessed as being at high risk of bias. Whilst in almost all trials (36, 90%), outcome assessors were aware the trial was taking place and in many (26, 65%) they were aware of the intervention received by the participant, because most outcomes were assessed as objective (30, 75%, Table 2) this lack of blinding was assessed as inconsequential (for outcome assessment).

Domain 5 bias in selection of the reported result: A large proportion of the trials (22, 55%) were assessed as at high risk of bias in the selection of the reported result, and this arose due to multiple reasons. For a sizeable number of trials (14, 35%) the primary outcome was not clearly defined, either because the outcome itself was not clearly defined (7, 17.5%) in any of the trial registration database, study protocol, or methods section of the main trial report, or, because the primary assessment time was not clearly defined (9, 22.5%). For a few trials it was not stated if the primary analysis would be adjusted or unadjusted for covariates (6, 15%). Almost all trials reported the scale the primary outcome would be measured on, and how any binary variables would be categorised, but some were assessed as not having a plan for how they would handle missing data despite having missing data (9, 22.5%).

 Reliability of independent assessmentsThe raw percentage agreement between the independent assessments were calculated for each signalling question, domain and overall risk of bias for each paper (Table 3 and Supplementary Table 8). For the overall assessment of each study the agreement was high (Gwet's AC: 0.92 95% CI: 0.85,0.99), but this varied across the different domains: agreement was 0.46 (95% CI: 0.20,0.72) for domain 1a (randomisation process); 0.59 (95% CI: 0.37,0.81) for domain 1b (identification and recruitment process); 0.85 (95% CI: 0.74,0.96) for domain 2 (deviations from intended interventions); 0.77 (95% CI: 0.62,0.92) for domain 3 (missing outcome data); 0.79 (95% CI: 0.64,0.95) for domain 4 (measurement of the outcome) and 0.44 (95% CI: 0.19,0.70) for domain 5 (selection of reported result).

Particular signalling questions which had strikingly low reliability included whether the allocation was concealed from the clusters at randomisation (0.41, 95% CI: 0.19,0.62); whether the selection of individual participants was likely affected by knowledge of the intervention (0.56, 95% CI: 0.36, 0.76); whether there were baseline imbalances across individual-level characteristics (0.53, 95% CI: 0.33,0.73); whether participants were aware of their assigned intervention (0.53, 95% CI: 0.33,0.74); whether proportions of missing data were similar across interventions (0.59, 95% CI: 0.40,0.78); as well as selection of reporting, for both the outcome (0.58, 95% CI: 0.36,0.79) and selected analysis (0.52, 95% CI: 0.30,0.74).