Introduction

The aim of non-inferiority trials is to show that there is no relevant loss of efficacy of the experimental arm as compared with the control.1 For this purpose, a non-inferiority margin is used to assess whether the experimental treatment is better than, or at least preserves the effect of, the active control.2–4

The non-inferiority margin is selected using one of four frameworks to define what fraction of the active comparator treatment effect the experimental treatment must be demonstrated to preserve: (1) indistinguishability, (2) reasonable comparability, (3) clinical acceptability considering other advantages, and (4) indirect demonstration of superiority over placebo.2–7 In the most stringent, indistinguishability approach, the non-inferiority margin is defined as the minimal clinical important difference (MCID) for the outcome. The MCID is the smallest change, meaningful to patients.8 However, sample sizes required to demonstrate indistinguishability are often infeasibly large. Therefore, trials may instead undertake the reasonable comparability approach, in which the non-inferiority margin is selected based on the level of uncertainty regarding non-inferiority of the experimental treatment that clinicians are willing to accept to proceed with incorporating the new treatment into clinical practice.

One treatment approach for ischemic stroke that is currently undergoing intensive investigation in randomized non-inferiority clinical trials is treating large vessel occlusions with endovascular treatment (EVT) alone (novel approach, EVT alone) compared with intravenous thrombolysis with alteplase (IVT) plus endovascular treatment (standard active treatment, IVT+EVT) in patients presenting directly at an EVT-capable center. None of the first six trials provided detailed clinical justification for the non-inferiority margins they selected 9–15 a common infelicity in design or reporting of non-inferiority trials.16 To interpret the results of these studies, it will be helpful to consider trial findings in the context of both the indistinguishability margin and the reasonable comparability margin for their functional independence endpoint (modified Rankin Scale score 0–2 at 3 months). A prior study has indicated that the indistinguishability margin value is 1.3%,17 but to our knowledge no study has yet been undertaken to define the reasonable comparability margin.

Only clinical judgment, not statistical analysis of past studies, can define what fraction of the active comparator treatment effect must be preserved for the experimental treatment to be considered reasonably comparable to the reference treatment. Establishing this value is critical to interpretation of the results of the EVT alone versus IVT+EVT trials. We therefore undertook an international survey study to characterize what experienced stroke clinicians would consider an acceptable level of uncertainty regarding hypothetical decisions on whether to skip or administer IVT before EVT in patients admitted directly to EVT-capable centers.

MethodsDesign

We designed a survey response framework for a hypothetical trial comparing an IV lytic drug followed by an endovascular intervention (standard of care) with endovascular intervention alone (experimental arm), in which a similar proportion of patients admitted directly to an EVT-capable center achieved functional independence at 90 days. Physicians were asked at what level of certainty they would feel comfortable skipping the IV lytic drug in clinical practice, considering these hypothetical trial results. The survey included 12 questions on three separate web pages. The first seven questions, together with the description and aim of the survey, were displayed on the first page and assessed baseline responder variables, including sex, training, geographic location, country, appointment level, annual mechanical thrombectomy volume of the center and time devoted to the care of patients with a stroke.

The eighth question was displayed on the second page and dealt with the non-inferiority margin or the perceived acceptable level of uncertainty using the following scenario:

‘In an acute stroke trial, patients with a large vessel occlusion are randomized to an IV lytic drug followed by an endovascular intervention (standard of care) or the endovascular intervention alone (experimental arm). The endovascular intervention without the IV lytic drug yielded the same number of patients achieving functional independence (modified Rankin Score 0–2) at 90 days after stroke, but the degree of certainty of this result is limited by trial sample size. This uncertainty implies that, while it is most likely that skipping the drug has no effect on the outcome of patients, it is also possible that it improves or worsens the outcome to some degree. There are several pathophysiological arguments in favor and against using the IV lytic drug before the endovascular intervention (current standard of care). How much certainty would you deem sufficient to skip the IV lytic drug and treat patients with the endovascular intervention alone in clinical practice? I would feel comfortable skipping the IV lytic drug if the best estimate is that endovascular intervention alone yields the same number of independent outcomes, there is also a possibility that skipping IV lysis increases the number of independent outcomes, and I can be highly confident that, in the worst possible scenario, no more than X out of 100 patients treated with endovascular intervention alone will fail to regain functional independence due to skipping the IV lytic drug.’

Response options to replace the X in the text above, were integer values without decimals ranging from 0 to 100 percentage points. For visual guidance six potential answer values were displayed graphically on the same page, specifically including 0 as a legitimate response option (online supplemental figure 1).

Three additional questions were displayed on the last survey page that elicited information about respondents' history of participation in three types of salient randomized-controlled trials. Specifically responders were asked to state if they participated as an investigator in trials which compared: (1) IVT versus placebo; (2) EVT+best medical treatment versus best medical treatment only; and (3) EVT alone versus IVT+EVT (online supplemental table 1). The final 12th question addressed management of administering IVT in patients undergoing EVT. The full survey questionnaire can be found in the online supplemental appendix. Data are available from the corresponding author on a reasonable request, together with a research plan proposal.

Distribution

Before distribution of the survey, there was an internal pilot phase with 25 participants giving feedback on the functionality and comprehensiveness of the survey. After incorporating the feedback, the survey was distributed among non-interventional and interventional neurologists, non-interventional and interventional neuroradiologists affiliated with a university hospital or large tertiary stroke care center around the globe. Email addresses and baseline variables were extracted from publicly available information, including institutional home pages and published curricula vitae. Physicians from all continents were invited, while being cognizant of non-interventional and interventional counterpoises. Sampling focused on major cities with a high volume of patients with a stroke while still maintaining sufficient continental representation. The initial invitation was sent out to 600 participants. We replaced 25 non-functioning email addresses of potential participants, and in 19 cases, participants were replaced, because no alternative functioning email addresses could be found. The final target population constituted 600 invited participants with functioning email addresses. After the initial survey invitation on July 26, 2021, three reminders were sent out to non-responders, with each reminder being sent after a 7-day period. The survey was closed and the database locked on August 23, 2021. Ethical approval and patient consent were not sought for this study owing to its nature of a proposed hypothetical trial.

Statistical analysis

Data descriptives are displayed as n/N (%) or median (IQR). Bar charts are used to display frequency of selected acceptable uncertainties, with inputs >10%, summarized as >10% for graphical purposes. Group comparisons were performed using standard univariable statistical measures (Mann-Whitney U test for ordinal variables, exact Fisher’s test for categorical variables). Association of respondents’ characteristics with selected acceptable uncertainty was additionally evaluated using logistic regression analyses. For logistic regression analyses, the acceptable uncertainty was defined as an independent variable and was split at different quartiles (primary analysis: median split, sensitivity analyses: 25% and 75% quartiles). Potential for response bias was evaluated comparing non-respondents and respondents, as well as comparing early and late responders (using a median split of respondent timing). Incomplete responses were rated as non-responders. Medical training was presented as a dichotomized variable encompassing interventional (interventional stroke neurologists and neuroradiologists) and non-interventional training (vascular stroke neurologists and diagnostic neuroradiologists). Goodness of fit for baseline variables was evaluated by comparing responses provided by the responders with those extracted from publicly available sources. κ Values were calculated to evaluate agreement. κ Values of 0.6–0.79, 0.80–0.90, and >0.90 were rated as moderate, strong, and almost perfect agreement, respectively, corresponding to 35–63%, 64–81%, and 82–100% reliability of the data. All tests are two-sided and a p value ≤0.05 was considered significant.

Results

There were 180 survey respondents (response rate: 30%), and 165 respondents completely answered all survey questions. Baseline characteristics of the respondents with complete answers are shown in table 1. Almost three-quarters were male, and there was an equal representation of interventional and non-interventional subspecialties. Most respondents were at a senior appointment level and had 10–50% of their care time dedicated to patients with a stroke. When asked about IVT management in patients undergoing mechanical thrombectomy, 61.8% stated they did not stop the full IVT dose unless there was a clear medical reason to do so, whereas others answered that they stopped IVT infusion if certain criteria were met.

Table 1

Respondents’ baseline characteristics

Representativeness

Goodness of fit for baseline variables were 0.96 (95% CI 0.91 to 1.00), 0.69 (95% CI 0.58 to 0.79), 0.96 (95% 0.92–1.00), 0.86 (95% CI 0.79 to 0.96) for sex, interventionalist versus non-interventionalist training, geographic location, and appointment level, respectively. There were differences in response rates among geographic regions, appointment level, and medical training. Junior and mid-career physicians were more likely to respond than senior physicians (40.7% and 43% vs 25.5%; p=0.004) and European participants were more likely to respond than participants from other geographic areas to the invitation (67.9% vs 32.1%, p=0.001). Similarly, those with medical training in vascular stroke neurology and interventional neuroradiology had better response rates than their counterparts in interventional stroke neurology and diagnostic neuroradiology (31.9% and 31.5% vs 19.3% and 13.4%; p=0.001; table 2). There were no differences between early and late responders (online supplemental table 2).

Table 2

Baseline comparison of complete against incomplete and non-responders

Physicians’ judgments of acceptable uncertainty

The distribution of respondent’s judgements of the acceptable value for uncertainty is shown in figure 1; the median value was 3 out of 100 patients, with IQR 1–5. The most chosen acceptable uncertainty was 5 out of 100 patients, with close to every third respondent (29.1%) choosing this answer. A sensitivity analysis excluding respondents who defined values >10 as acceptable uncertainty (n=6) did not change median or interquartile range (3, 1–5, respectively). When limiting the analysis to physicians with senior-level appointment (n=126), acceptable uncertainty had the same values for median and interquartile range (3, 1–5, respectively).

Figure 1

Distribution of respondents’ acceptable uncertainty values. There was a right skewed distribution with the median response being 3 (IQR 1–5), indicating that most responders would skip the intravenous therapy and treat patients with endovascular intervention alone if effect point estimates were equal; either treatment could be a little better than the other; and, in the worst possible scenario, 3% of the patients receiving endovascular treatment alone would fail to regain functional independence due to skipping the intravenous therapy. Most respondents (29.1%) selected 5% (mode) as acceptable uncertainty.

Distributions of acceptable uncertainty values by strata of sex, trial participation, training, and percentage of dedicated care time for patients with a stroke, are shown in the supplementary material (online supplemental figures 2-11). The median value of selected acceptable uncertainty values and IQR of all subgroups is displayed in online supplemental table 3. The most chosen answer was 5 out of 100 patients in all subgroups, except for respondents participating in IVT versus placebo trials (most chosen uncertainty: 1 out of 100 patients) and those with non-interventional training (most chosen uncertainty: 3 out of 100 patients).

On multivariable logistic regression median split analysis, interventional training was associated with higher values for acceptable uncertainty (aOR 2.20, 95% CI 1.06 to 4.67), while senior appointment and annual number of mechanical thrombectomies in center >200 tended to be associated with lower uncertainty margins (aOR 0.27, 95% CI 0.05 to 1.09; aOR 0.57, 95% CI 0.25 to 1.25, respectively), as presented in figure 2. The directions of most associations were comparable when using first and fourth quartile splits, with participation in IVT versus placebo trials and participation in EVT versus IVT+EVT trials being associated with selecting lower acceptable uncertainty values on first and fourth quartile split analyses (online supplemental figure 5).

Figure 2

Forest plot showing effect of respondent characteristics on acceptable uncertainty values. Logistic regression analyses with dichotomized (median split) acceptable uncertainty margins defined as dependent functional outcome, were performed to evaluate associations between baseline characteristics and lower/higher chosen acceptable uncertainty margins. ORs >1/<1 indicate association with higher/lower acceptable uncertainty values, respectively. IVT, intravenous thrombolysis; MT, mechanical thrombectomy.

Discussion

This survey found that physicians would generally feel comfortable skipping intravenous thrombolysis before mechanical thrombectomy in directly admitted patients at different degrees of certainty. For more than half of survey physicians, this certainty threshold was met if randomized trial data suggested that the number of patients achieving functional independence at 90 days was similar between both treatment groups, and one could be sufficiently sure that—in the worst case scenario—no more than 3/100 patients would not achieve functional independence at 90 days, owing to skipping intravenous thrombolysis. Moderate variance was noted among physicians as to what was considered an acceptable uncertainty, and there was weak evidence that uncertainty value selections differed with respect to training, case volume, and prior participation in specific clinical trials.

The findings of the current study are consonant with prior investigation. As expected, this study’s value for acceptable uncertainty when assessing reasonable comparability (3%) is modestly larger than the value for the MCID when assessing indistinguishability (1.3%) found in a prior study using a superiority design scenario of a novel and safe neuroprotective agent.17 Although the sample size required to demonstrate non-inferiority using a margin based on the MCID is unattainably large (18 205 patients per group for a study with 80% power and 95% confidence), the sample size required to demonstrate non-inferiority based on a reasonable comparability margin is less extreme, though still sizeable (3419 patients per group).

The finding of a 3% reasonable comparability margin provides a useful comparison for non-inferiority margins employed in the six completed randomized trials of EVT alone versus IVT+EVT.9–15 Three of these trials used absolute risk difference margins12 14 15 and three relative risk difference margins.9–11 13 In the three trials using absolute risk difference, non-inferiority margins for functional independence at 3 months were defined as three to four times higher (ie, 10% and 12%)12 ,14 15 than the median reasonable comparability margin found in this survey. The chosen absolute non-inferiority margin of 10% and 12% in those trials is equal to a preserved fraction of 60% and 51% of the endovascular treatment effect in the SWIFT PRIME trial, and 51% and 41% in the HERMES meta-analysis, respectively.12 14 15 18 19 The other three trials, which used relative risk non-inferiority margins, were of generally similar magnitude to those used in the absolute risk margin trials.9–11 13 Importantly, each of the completed randomized controlled trials is thus not powered to fully assess reasonable comparability at a 3% margin on their own. However, if individual trials or a meta-analytic aggregation of their findings, narrows the CI to below a 3% margin, then reasonable comparability can be demonstrated.

The current study employed survey question wording designed to elicit physician judgment of reasonable comparability (acceptable uncertainty) rather than indistinguishability (MCID) of treatments. In a prior survey assessing the MCID, the survey query was posed in a superiority frame, asking physicians to choose the smallest difference in outcomes that would lead them to use a simply delivered drug therapy in clinical practice.17 In addition, the query asked physicians to consider treatment effects within a denominator of 1000 patients in order to allow greater resolution in specifying small values. In contrast, in the current study, the survey query was posed in a non-inferiority frame, asking what uncertainty level physicians would judge sufficient to consider a less complicated rather than more complicated treatment strategy (ie, skipping IVT before EVT). With this approach, physicians indicated the maximally acceptable potential risk that patients may have a poorer outcome due to their treatment decision, although the framework of the survey implies that their decision may also be equal to, or more beneficial than, the reference treatment. In addition, the query asked physicians to consider treatment effects within a denominator of 100 and specifically permitted ‘0’ as a possible answer. This allows for the option that physicians will only skip IVT if there is virtually no uncertainty around the interchangeability of treatments, which would mean that the lower bound of the 95% confidence would just cross or not cross 0%, formally representing superiority of the EVT alone approach.

In the present study, physicians trained in non-interventional specialties selected more stringent acceptable uncertainty values than did physician interventionalists. This difference may in part reflect each group’s familiarity with, and allegiance toward, the treatment modality they themselves deliver: IVT therapy for non-interventionalists and EVT for interventionalists. Moreover, it may reflect the thought process of interventionalists administering dual or single antiplatelets during the intervention in cases which require cervical stenting. In the setting of heightened risk of hemorrhage, they may tend to accept a higher level of uncertainty. Also, interventionalist physicians may generally be characterologically more accepting of risk than non-interventionalists.20

Limitations

This study has some limitations. First, selection of participants was based on publicly available institutional website information, publications, and participation in trials, rather than a random sample of all stroke physicians. The response rate was moderate, and senior physicians and European physicians were over-represented among respondents compared with their numbers in actual practice. Second, as with all surveys, this survey was subject to anchoring, centrality, and framing bias. Figural examples of response options were included to maximize understanding, and examples can exert anchoring bias, but the lack of response peaks at example values suggests this did not occur. Third, use of a reference group size of 100 might have modestly increased risk taking by contextual group size differences, as it has been shown that risk-seeking behavior is generally greater if group sizes are presented in small numbers in experimental life–death decision problems.21 22 Fourth, we were unable to assess the actual knowledge of, and familiarity with, non-inferiority trials of the survey participants, which might have influenced their decision and comprehension of the survey question. Last, there are several key points considered in a non-inferiority trial, including disease prevalence, practicality, and cost-effectiveness, which were not fully incorporated into the given scenario. We intentionally kept the factor of cost out of the equation, since economic considerations are probably affected by location and time. As such, it could introduce unnecessary heterogeneity, limit generalizability of the study results, and might have confounded clinical judgment.23

Conclusion

This study found that international expert stroke non-interventional and interventional physicians would consider direct EVT reasonably comparable to IVT+EVT and be willing to treat patients admitted to comprehensive stroke center with direct EVT at different certainty thresholds. Most physicians would treat with direct EVT if the following criteria were met: randomized trials showed the same point estimates for functional independence outcome, and the confidence intervals excluded the possibility that no more than 3/100 patients would not regain functional independence due to skipping the drug. Non-interventional physicians were modestly more conservative than interventional physicians in the degree of uncertainty they would find acceptable. This value for acceptable uncertainty based on survey data from a predominantly European group of international stroke experts may be useful for selecting the non-inferiority margin when meta-analysis is undertaken pooling data from all completed trials.

Data availability statement

Data are available upon reasonable request. Data are available from the corresponding author upon a reasonable request, together with a research plan proposal.

Ethics statementsPatient consent for publication

Not applicable.

Ethics approval

This study does not involve human participants.