The SAGe tool is designed for use in formal safety review meetings (e.g., Safety Information Review Committees, Global Labelling Committees, Safety Evaluation and Risk Management activities); however, at AstraZeneca, use of the tool is also recommended as best practice when evaluating a safety signal that has been identified through routine surveillance or receipt of a signal from an external source (e.g., a health authority).

While the conceptual framework of the tool is predicated on the tripartite categorisation of data analyses noted previously (aggregate safety data, plausibility data, and case-level data), the methodology of the tool requires a hierarchy of evidence types for each category (Tables 1, 2, 3). This acknowledges that certain evidence types are more compelling than others [7]. The challenge lies in delineating the details and ranking of this hierarchy, a step we have taken based on our experience of AstraZeneca’s history of safety signal assessments.

Assessing the robustness of available data is, to some extent, open to interpretation and requires expert clinical judgement. When developing the tool, we referred to published literature pertaining to the critical appraisal of study design [8, 9]. Subsequently, the adoption of evidence levels (Levels A, B, C, and D) to score the robustness of data related to safety signals was predicated on an internally generated operational definition (i.e., built on practical experience) that one example of Level A data from any one of the three categories (Tables 1, 2, 3) would typically be sufficient, alone, for a positive safety signal causality assessment. Similarly, the lowest level of the SAGe evidence range, Level D, was operationally defined as data that, while worth documenting and evaluating, do not, in isolation, lead to a positive causality assessment. Even if Level D evidence is present in all three SAGe categories, this is insufficient for a positive causality assessment; inclusion of Level D evidence is to ensure thoroughness and transparency of the assessment. Level B evidence in a category may be sufficient to support a positive causality decision, particularly if combined with evidence from other categories. Level C evidence is typically insufficient for a causality decision unless clearly and consistently present across all three SAGe categories (or unless combined with a higher letter-grade evidence level). A not-applicable (N/A) grade is allowed for a category with no qualifying data.

Only the highest evidence level for a category is scored. For example, if a category has qualifying data from both Levels B and C, only Level B is acknowledged in the final analysis. The additional existence of the Level C evidence does not fundamentally strengthen the evidence in our methodology.

The management of negative (i.e., contradictory) evidence within a category is managed on a case-by-case basis. In general, higher-scoring evidence outweighs the lower-scoring evidence; for example, aggregate Level A evidence suggesting positive causality overrides aggregate Level B evidence against causality. However, the end result is that final grading for the aggregate category may be modified, e.g., Level A may go down to Level B or C, depending on medical judgement.

We also allow the algorithmic derivation of a quantitative signal strength score based on various combinations of evidence levels from each of the three categories (see electronic supplementary material). This quantitative score is visualised by two reciprocal bars representing ‘signal strength higher’ and ‘signal strength lower.’ These numerical estimates of signal strength can be used to support decisions regarding signal validation and updates to the CCDS.

Using the SAGe tool methodology, AstraZeneca safety teams for both drugs in development and marketed products are expected to organise and present a summary of their safety signal evaluation in a structured fashion. For each of the three categories, data are scored (i.e., by Levels A, B, C, or D) and concisely summarised. Subsequently, all three of the highest letter-grade evidence level scores from each category are combined into a single graphic, which then serves as a shorthand summary of safety signal strength for a topic (see examples in Figs. 1, 2, 3). For assessments without data that warrant a letter-grade level evidence score in each category, N/A is used. Situations in which conflicting evidence is presented within a category are generally resolved algebraically, with higher-scoring evidence outweighing the lower-scoring evidence, as described in the previous section.

Example of a marketed product safety signal from a case report series in literature

Example of a peri-submission product safety signal from an imbalance in aggregate pooled data

Example of a phase I product safety signal related to plausibility from preclinical findings qualifying as an important new safety issue

Following application of the SAGe tool to a safety signal causality assessment, AstraZeneca project safety teams incorporate the output from the tool into their causality recommendation and typically take this recommendation to a formal safety review board for consideration and decision making. As at the signal evaluation level, decision making by the safety review board incorporates use of the SAGe tool to support the medical and scientific evaluation process. Medical judgement (i.e., global introspection) remains our standard for decision making.