Benefit-risk method selection

The B-R methods represented in the case studies below have been taken from a previous rapid systematic review which outlined available B-R methods for use in publicly funded RCTs [17]. From the initial list of 96 methods, those deemed not relevant (n = 29), not appropriate due to only considering 1 outcome (n = 5) or requiring extra data to that typically collected in an RCT (n = 45) are not taken forward. The remaining 17 methods which could be potentially useful in this scenario spanned across different method types (as defined by the PROTECT group [14]) including descriptive or quantitative frameworks, trade-off metrics and visual methods (Table 1). This list is not meant to be exhaustive, but rather a shortlist of potential methods as a starting point for those looking to improve the reporting of adverse events.

Table 1 Shortlist of appropriate benefit-risk methods for use when presenting adverse event data in RCTs

Many of the visual methods are not specific to benefit-risk methods, and as comprehensive research into the optimum visual presentation of adverse events is available [21], these methods will not be considered in further detail. One method was selected for illustration purposes from each of the remaining method type groups. The authors selected the methods based on their perceived usefulness to illustrate improved adverse event reporting and well as those that are commonly used within the literature. Methods selected were the Benefit-Risk Action Team (BRAT) Framework, net clinical benefit (NCB) and the Outcome Measures in Rheumatology (OMERACT) 3 × 3 table.

The BRAT framework will be applied to both case studies whereas NCB will be applied to case study 2 and OMERACT table to case study 1 only. This is due to the primary outcome data type which was continuous for case study 1 and binary for case study 2. The OMERACT method requires categorisation into three to represent the deemed strength of the benefit which was possible on the primary outcome due to the summaries provided by the case study 1 results. This categorisation should be straightforward for the presence of certain adverse events but will require the categorisation of any outcomes such as pain level into categories that is considered none/minor, major and (near) death. However, dichotomisation of this outcome would have required potentially further clinical knowledge. As NCB is related to the probability of an event occurring this is directly applicable for the presence of adverse events and especially easy when the primary outcome is also binary as in case study 2. To note here, a potential solution to dichotomise continuous outcomes would be to use a distributional approach as recommended for continuous adverse events [29].

Benefit-risk methods descriptionBenefit-Risk Action Team (BRAT) Framework

The Benefit-Risk Action Team (BRAT) Framework was first reported in 2011 to improve the expert judgement of clinical decision-making by being more ‘transparent, rational and defensible’ [24]. This framework was created to provide a set of guiding principles that were suitable for use by all stakeholders to complete a benefit-risk assessment and concluded with a six-step process as follows:

1.

Defining the decision context

2.

Identifying the benefit and risk outcomes

3.

Identifying data sources

4.

Customising the framework

5.

Assessing the relative importance of the outcomes

6.

Displaying and interpreting the key benefit-risk metrics

As we can assume steps 1–5 are completed within the design and conduct of an RCT, this paper will focus on step 6—displaying the results; however, the value tree in step 2 will be presented for completeness.

For reporting, the BRAT framework recommends the use of a key benefit-risk summary table which they state has been ‘designed to allow users to readily grasp the major issues underlying a benefit-risk assessment’ [24]. This table provides both the key benefits and risks separately and alongside each other which allows direct comparison but no formal calculation of trade-off between outcomes is completed. Benefits in this case are defined as favourable events and risks are any unfavourable events potentially associated with the treatments. The structure of the table, at a minimum, shows the event rates for both treatment groups, the difference (on the same scale for each outcome if possible) and the variability of this difference. There is the ability to add additional information such as stakeholder preferences if it is available and useful.

The creators provided a worked example of the BRAT framework [30] which gave further details and key considerations. In this example, all of the outcomes are binary and therefore can all be considered on a risk difference scale (adjusted to per x patients for comparability). However, continuous outcomes are common within RCTs. The authors of the BRAT framework originally stated continuous measures can be included if they can be dichotomised [30] but due to the loss of information with this [31], we believe these can be included without dichotomisation by using a standardised effect size measure for comparability and this has been implemented within the case studies. A pilot programme using the BRAT framework in practice was completed in 2012 and found it to be effective and flexible to different circumstances with the key benefit-risk summary table to be a particularly attractive element [32].

Net clinical benefit (NCB)

The net clinical benefit (NCB) framework was developed in the context of individual patient decision-making [33] but can also be used to assess the average benefit-risk balance of treatments among a trial population. It is a quantitative framework which addresses the need to weigh up all the benefits and risks of treatment simultaneously. When applied to a particular treatment, the method generates a metric, the expected net clinical benefit, calculated as the expected overall benefit of treatment less the expected overall harm.

Within the NCB framework, benefits and harms must be characterised in terms of probabilities, this is straightforward to do for binary events (see SANAD II case study) whilst for non-binary events, additional calculations are required. Furthermore, in order to integrate the benefits and harms into a single metric, they should be expressed on a comparable scale; in general, this will require weighting each benefit and harm in the NCB calculation in proportion to its impact, using elicited utilities or quality-of-life data, for example. The general NCB equation then becomes:

$$\text= \sum___-\sum___$$

where B is the set of benefits, H is the set of harms, \(_\) is the weight associated with outcome i and \(_\) is its probability of occurrence.

To compare treatments in terms of their overall benefit-risk profile, the difference in expected NCB between each treatment should be calculated—treatments with higher expected NCB have a better benefit-risk balance on average.

In simple cases with small numbers of benefits and harms and/or where all events are of similar impact, it may be possible to determine which treatment has a higher NCB without the use of explicit weightings, but in such cases, caution should always be taken to ensure the conclusions are robust.

Outcome Measures in Rheumatology (OMERACT) 3 × 3 table

In rheumatology RCTs, researchers realised the need for a simple method that could consider both the benefits and risks [27]. This method builds on the principle of the threes method [34] whilst remaining more straightforward than some of the more complex benefit-risk methods. The method simply categorises patients into whether their overall benefit was none/minimal, major or (near) remission. The same is then considered for the patient’s overall harm being none/minimal, major, or (near) death. A 3 × 3 table is created using these two categories to show the percentage of patients in each subset. The table is colour-coded to represent the benefit/risk with black indicating no benefit with high risk and white showing high benefit with no risk and a mixture of red, orange and yellow used for the cells in between. A separate table is created for each treatment which allows direct visual comparison, but there is no formal numerical comparison between treatment options.

The strength of this method is in the trade-off of outcomes at the patient level, instead of the population level, which is recommended for benefit-risk assessments [35, 36]. The authors recognised the drawbacks of simplification that occurs when only three categories are used but highlight that it avoids a complex weighting process whilst still being able to group similarly important outcomes together. At a glance, it allows the reader to see whether patients are benefiting with/without harms, and therefore, a reflection of the overall benefit-risk summary of the treatment. Although criticism states that as this is over two tables it can be hard to directly compare [26].

Case studiesOption-DM

Option-DM [37] was an RCT (n = 140) for patients with diabetic peripheral neuropathic pain as four different treatment drugs were recommended so a direct comparison of these was required. This was a crossover trial which randomised equally to six different treatment pathways. For simplicity, the data from just two of the treatment pathways (monotherapy of amitriptyline and combination therapy of pregabaline followed by amitriptyline) will be used for demonstration purposes. The primary outcome was the 7-day average daily pain at the end of each pathway, although this is a continuous outcome the team have provided different dichotomisation options of a 30% pain reduction, a 50% pain reduction or a pain score of less than 3 (low pain).

The RCT showed similar efficacy across the treatment pathways but an improvement in change in pain score for those on combination therapies compared with monotherapies (mean change 1.0 [SD = 1.3] vs 0.2 [SD = 1.5]). Secondary patient-reported outcomes however were only presented as continuous outcomes. Adverse events of dizziness, nausea and dry mouth were key findings from the RCT as described in the abstract and in total 16 different adverse events were reported tabularly.

Current display of adverse events

In the main publication [37], reporting of adverse events was restricted to those that were treatment-emergent and occurring in more than 5% of patients. These are summarised in tabular form as the number (%) of patients experiencing each event, together with P-values from global chi-squared tests comparing the rates of adverse events between therapies. Table 2 shows the summaries of the number of adverse events for the two treatments of interest, monotherapy of amitriptyline and combination therapy of pregabaline followed by amitriptyline.

Table 2 Summary of the number of adverse events in the Option-DM RCT, reproduced from the results paper [37]

The more detailed study report [38] summarises a complete list of adverse events, both as the number of events and as the number of patients experiencing an event. In addition, adverse events are presented by treatment phase (< 6 weeks or > 6 weeks), as a subset of those that are moderate or severe and as a subset of those that are moderate or severe and related to study treatment. Bar charts showing the mean number of days affected by each adverse event per 1000 days follow-up are reported, together with tabular summaries of average weeks affected per person and days affected per 1000 days follow-up. The number of serious adverse events and the number of patients experiencing serious adverse events are also summarised by treatment pathway (any serious adverse event and the eight most common serious adverse events), although few occurred in the study (n = 31).

BRAT value tree and table

The value tree (Fig. 1) clearly outlines the key benefits (favourable outcomes) and risks (unfavourable outcomes) from the RCT which will go into the overall benefit-risk assessment of the treatments. The favourable outcomes within the RCT are quality of life as well as mood and sleep which are measured using a range of patient-reported outcome measures. The primary outcome of a 7-day average of daily pain is included as an unfavourable outcome in the value tree, it is assumed this will be improved by the treatment and pain will be reduced. Additionally, the most common adverse events were included as risks. The outcomes identified in the value tree are also used within the BRAT summary table (Table 3) which shows the results of both treatments and the treatment difference (with confidence intervals).

Fig. 1

BRAT value tree for the Option-DM RCT

Table 3 BRAT summary table for the Option-DM RCT

Using this summary table, the reader can easily see that there is no discernible difference between the treatments on any of the benefit categories. However, a difference is apparent in some of the adverse events with two particularly large differences (dizziness and oedema) in favour of the monotherapy amitriptyline.

OMERACT 3 × 3 table

Application of the OMERACT 3 × 3 table required a decision of which outcomes represent none/minimal, major and near remission/death for the benefits and risks. The categories chosen based on the information contained in the RCT outputs are outlined below. For simplicity, only the primary outcome of reduced pain has been chosen for the benefit and categorised by a percentage improvement. However, this can be extended to include multiple outcomes as a composite using patient-level data.

The following are the benefits:

(Near) remission: pain reduction of more than 50%

Major: pain reduction of more than 30% but less than 50%

None/minimal: pain reduction of less than 30%

The following are the risks:

(Near) death: serious adverse event

Major: moderate or severe treatment-related adverse events (as defined by the trial team)

None/minimal: no adverse events or minor adverse events

As we do not have access to the patient-level data for the Option-DM RCT, three versions of the 3 × 3 table have been created (Tables 4 (a–c)) which all indicate different potential patterns that the data could show given the summary statistics we have access to. The purpose of completing three versions is to demonstrate the additional information that can be provided by using this method and how this could influence opinions of treatment. The total column has been taken from the data presented in the outputs and just the cell data created. Table 4 (a) represents a situation where there was no discernible relationship between the benefits and risks and the percentages are proportional to the total columns. Table 4 (b) shows a skew towards higher benefits with the treatment being related to also having higher risks. Finally, Table 4 (c) suggests two subsets of patients with a skew towards some with high benefits having lower risks and vice versa with those patients with higher risks seeing lower benefits. Clearly, this will create three very different conclusions to the treatment benefit-risk balance.

Table 4 OMERACT 3 × 3 table for the OPTION-DM RCT for those on combination therapy pregabaline followed by amitriptyline (n=107) where the responses are (a) proportional, (b) skewed towards higher benefits related to higher risks and (c) skewed towards two groups of patients with high benefits/low risks and high risks/low benefits SANAD II

The SANAD II study [39] was a non-inferiority RCT for patients with newly diagnosed focal epilepsy (n = 990). Patients were equally randomly allocated to receive either lamotrigine (control treatment), levetiracetam (treatment 1) or zonisamide (treatment 2), which are both licenced for use but longer-term effectiveness data were not available.

The primary outcome, assessed on a non-inferiority basis, was time to 12-month remission (non-inferiority margin hazard ratio (HR) of 1.329, 10% absolute difference). Levetiracetam was found not be non-inferior to lamotrigine (intention to treat analysis of time to 12-month remission HR = 1.18; 97.5% CI [0.95–1.47]). However, zonisamide was found to be non-inferior to lamotrigine (HR = 1.03; 97.5% CI [0.83–1.28]).

Analysis of adverse events showed that fewer participants with reactions for those in the lamotrigine arm (n = 108/328, 33%) than both the levetiracetam (n = 144/330, 44%) and zonisamide (n = 146/324, 45%) arms. Lamotrigine was superior in the cost-utility analysis and therefore, despite being clinically non-inferior, zonisamide was not recommended by the clinicians due to the increased adverse reactions identified and lamotrigine continues to be recommended as the first-line treatment. In this project for simplicity, we will only be reporting two of the treatments (lamotrigine and zonisamide).

Current display of adverse events

Within the results article for the SANAD II study [39], the number of adverse events and number and percentage of patients experiencing adverse events were reported by MedDRA system organ classification in the main publication as shown in Table 5. In the article's supplementary appendix, a more granular breakdown was given by MedDRA preferred term, and a separate table summarised only those adverse events classified as severe. A listing of all serious adverse reactions (SARs) was provided in the full study report [40] which included the seriousness, severity, expectedness, suspected relationship to the study drug, action taken and outcome. No formal statistical comparisons between treatment arms were performed.

Table 5 Summary of adverse reactions in the SANAD II RCT, reproduced from the results paper and HTA report [39, 40] BRAT value tree and table

As this is a non-inferiority trial, the BRAT value tree (Fig. 2) may look slightly different as one of the outcomes, in this case seizure remission, is expected to be similar instead of better/worse. This is the main favourable outcome within this RCT and the adverse events are the main unfavourable outcomes.

Fig. 2

BRAT value tree for the SANAD II RCT

The adverse events have been categorised by the clinicians in the RCT and these categories have been used in this summary. The difference in percentage is provided along with the respective 95% confidence interval as this was the most consistently reported metric across all of the outcomes in the summary table (Table 6). The table shows the seizure remission rate has a 5% point benefit for zonisamide; however, the confidence interval is fairly wide and includes zero. The rates of adverse events are all higher in the zonisamide arm with differences of up to 9% where the confidence interval is much narrower and does not include zero in many cases.

Table 6 BRAT summary table for the SANAD II RCT Net clinical benefit

The net clinical benefit for zonisamide relative to lamotrigine is derived from the probability differences shown in the final column of Table 6. Using the formula given earlier, expected NCB is calculated as a weighted sum of the probability differences for benefits minus a weighted sum of the probability differences for risks, where the weights reflect the relative clinical importance of each outcome. Assuming for simplicity that all benefits and risks are weighted equally allows us to omit the weights from the formula and thus calculate the expected NCB for zonisamide relative to lamotrigine as:

$$\text= 5-\left(9.4+2.4+6.9+0.4-0.8+3.1+4+0.6\right)$$

Since the expected NCB is negative, we conclude that on average lamotrigine has a more favourable benefit-risk balance than zonisamide, under the assumption that all benefits and risks are weighted equally. Should individual weights want to be included within the calculation of the NCB, further data would need to be collected from key stakeholders. An example can be found by Holmes et al. [41] where a discrete choice experiment was completed to estimate patient preferences in a related trial.