All databases were searched by NT on the 14th of January 2022. The ISRCTN webpage identified 406 trials, the NIHR Funding and Awards Library identified 113 trials with the NIHR Journals Library finding an additional ten. After the removal of duplicates, there were 477 records to be screened.

At screening, 362 trials were excluded with the two most common reasons being non-UK (n = 210) or industry (n = 94) funded. Five of the trials were excluded as they were bolt-on studies to an original NI trial, for each of these it was checked the original trial had been included. During extraction, one further entry was removed as it was a programme grant which intended to include a pilot NI trial but did not progress to this stage. After exclusions, 114 NI trials were left to have their information extracted. Full screening numbers can be found in the flowchart in Fig. 1.

Flowchart for the database search and screening process

The characteristics of the trials included in the review are shown in Table 2. On average, the NI trials had a target sample size of 720 (IQR: 393–1204) participants although this ranged greatly with some large, population-based trials (max = 360,000 participants). Such sample sizes have resulted in expensive trials, with the median trial costing £1.77 million (IQR: 1.34–2.32) and on average, lasting 5 years (IQR: 4.0–6.8). As would be expected, the vast majority (101/114, 89%) were funded by the NIHR and just over half of these (58/101, 57%) were through a researcher-led call, i.e. no specific topic or suggestions were provided by the funder for the focus of the research. The included NI trials were at a range of stages, with most being either in the recruitment phase (39/114, 34%) or having already been completed (41/114, 36%).

The most common health categories were cancer (21/114, 18%) and infection (20/114, 18%). For the cancer trials, many of the rationales for completing a NI trial were based around the intensity and toxicity of cancer treatments whereas the rationale for infection NI trials was often the issue of antibiotic resistance. Drug treatments were the most common (86/250 treatments used within the trials, 34%) followed by surgical interventions (46/250, 18%).

Figure 2 shows the increase in NI trials over time by presenting the year the NI trial started. The use of non-inferiority as a term is expected to increase after the publication of the NI CONSORT Extension [34] and its update [12] which recommended specific use of the term. Nevertheless, the large increase in numbers still suggests an increase in popularity.

Starting year of NI trials included in the review (n = 114)

Most trials, 56% (64/114), had a binary primary NI outcome which has resulted in half (51%, 58/114) of the trials using a difference in absolute proportions, i.e. a risk difference, as the NIM (Table 3). The average absolute proportion difference used for the NIM (n = 58) was 8% (IQR: 3–10%), when calculated on a relative scale considering the control group rate, this represents a median of 20% (IQR: 9–39%) relative change. The standardised mean difference of continuous outcomes (n = 30) found a median value of 0.35 (IQR: 0.26–0.43), deemed to be a small to medium effect size [35].

The assumed difference between the treatment groups that were used within the sample size calculation was only reported in 32% (37/114) of the NI trials. When it was reported, most of the cases (81%, 30/37) assumed a difference of zero. Of the remaining seven, five trials assumed the intervention to have a slightly worse outcome with only two of the trials assuming the intervention would be slightly better. This assumed difference was not reported in 47% (54/114) of cases, despite it being essential to the replication of the sample size calculation.

There was a justification presented for the chosen NIM for 54% (62/114) of trials within the review, however, 21% (24/114) of the trials did not provide any justification at all despite sufficient documentation being available where this information should be reported (Table 4).

Statistical considerations were present in 21% (13/62) of justifications with most of these, however not all, being related to confirming superiority over placebo. Clinical considerations, i.e., confirming the difference is clinically unimportant, were much more prevalent being mentioned in 79% (49/62) of the justifications. This was often through consulting clinicians (30/49) but it was also common to see patient input included (20/49). The recommendations for considering both statistical and clinical considerations have only been implemented in 6 of the 62 trials which have a justification (10%).

In terms of primary analysis populations (Table 5), 43% of trials (49/114) selected to use intention-to-treat (ITT) only, with the next most common being both ITT and Per Protocol (PP) (15%, 17/114). Per protocol definitions were not assessed for consistency between trials and were identified by author-declared PP analysis. There was a small proportion (6/114, 5%) that did not specify the analysis population that they were using.

The significance level, summarised in Table 5, shows most NI trials and a 2.5% one-sided alpha (or equivalently 5% two-sided) within the sample size calculation (54%, 61/114), as recommended. However, it was not uncommon for a one-sided 5% alpha (25%, 29/114) to be used which increases the type I error within the trial. Finally, 90% power was used in 50% of the trials (57/114) with the less stringent 80% power used in 24% (27/114) of trials.

A range of added benefits were described within the review (Table 6), the most common being safety (28%, 32/114), typically represented by adverse events associated with the treatment. Patient-reported outcomes, convenience, cost and clinical outcomes were all also mentioned as the benefit of testing the new treatment.

Overwhelmingly, the main benefit of the new treatment was directly for the patient taking the treatment (61%, 70/114) for example with improvements in side effects or convenience of taking the treatment. However, in other cases, there was a benefit to the health service (25%, 28/114), e.g. reduced costs, or for public health, e.g. a reduction in antibiotic use across the population. All but one trial with enough information defined the added benefit expected by the treatment, however, 19 (/114, 18%) of all NI trials did not define a superiority outcome which would allow the evaluation of this added benefit within the trial.

Co-primary outcomes which included one NI outcome and one superiority outcome were used in just 18% (21/114) of the trials. A further 14/114 (12%) trials considered the statistical power for the superiority outcome even though they were not co-primary outcomes. Within the justification for the chosen NIM, the mention of the added benefit was present in 23% of cases (14/62). Many of these cases state the benefit in a general, unspecified manner, but some have quantified both the NI and superiority required for the new treatment to be deemed acceptable.