Study Selected for Demonstration

The study selected from the EU PAS Register® for the RSA-RWD Framework demonstration is titled, “Comparing the Estimated Risk of Hip Fracture Among Subjects Exposed to Tramadol as Compared to Subjects Exposed to Codeine” (EUPAS36038). Results of the completed study were published by Voss, et al. in 2022 [7]. The research plan was motivated by a previous publication by Wei, et al. (2020) [8] that reported an increased risk of hip fracture among new users of tramadol versus codeine. Voss, et al.’s study aimed to re-assess this relationship after addressing limitations of the prior publication (e.g., improved propensity score methods, use of morphine equivalents to estimate the exposure). The following sections provide a comprehensive assessment of the Voss, et al. study protocol against the RSA-RWD Framework. For each consideration for rapidity in the RSA-RWD Framework, we’ve outlined several identified enablers that were either implemented by the study team or could have allowed the research question to be addressed more rapidly.

Consideration 1: Alternatives to a Full Protocol

Development of a fully specified and executable study protocol for safety signal assessment involves a complex process of document creation, methodological considerations, organizational alignment, and approval. As an opportunity for rapidity, the RSA-RWD Framework describes that a simplified, alternative RWD analysis plan [9] may be used.

Identified Enabler 1: Use of an Abbreviated Specification

The selected Voss, et al. study describes the key design elements and planned analyses using a full protocol consistent with the requirements set forth by the European Medicines Agency (EMA) (Guideline on Good Pharmacovigilance Practices Module VIII [10] and Addendum I [11]), Food and Drug Administration (FDA) (Postmarketing Studies and Clinical Trials—Implementation of Sect. 505(o)(3) of the Federal Food, Drug, and Cosmetic Act [12]), and the Pharmaceuticals and Medical Devices Agency (PMDA) (Guidelines for the Conduct of Pharmacoepidemiological Studies in Drug Safety Assessment with Medical Information Databases [13]). In Fig. 4, we highlight the critical components that would be described if the TransCelerate Alternative to a Full Protocol (ATFP) Template for rapid signal assessment (RSA) was implemented.

Fig. 4

Summary of simplified protocol elements

Consideration 2: Data Source Identification

Identifying a fit-for-purpose RWD source for safety signal assessment involves considering whether the database includes variables essential to answer the research question and can be accessed within an expedited timeline. As an opportunity for rapidity, the RSA-RWD Framework describes that establishing a readily available RWD catalog with descriptions of accessible RWD sources can accelerate selection of appropriate data sources.

Identified Enabler 1: Compiling a Repository with Fit-for-Purpose Databases and Database Catalog

The selected study considered 6 RWD sources and ultimately chose 4 for analysis. Direct communication with the study investigator confirmed use of a pre-existing repository of in-licensed data sources. Among the total of 6 RWD sources considered, a single data source was selected to replicate the original publication. Outcome phenotype performance (i.e. sensitivity, specificity, positive and negative predictive value) was assessed in the additional 5 RWD sources and 3 of 5 data sources showed acceptable performance. The 3 sources with acceptable performance and the single source chosen for prior study replication were ultimately selected for further analysis. Compiling such a repository and a catalogue with detailed descriptions, metadata, and characteristics of each database as an overarching strategy allows investigators to quickly identify RWD sources, meet research needs and proactively avoid delays associated with assessing fitness for the purpose of new RWD sources.

Identified Enabler 2: Utilize Tools to Quickly Generate Feasibility Counts and Evaluate Data Sources

Consistent with the suggested rapidity best practices, the selected study assessed feasibility counts in all considered databases using a standard tool (CohortDiagnostics [14]). In addition, performance of the outcome phenotype was evaluated in each database using a tool (PheValuator [15]).

Identified Enabler 3: Prioritize RWD Data Sources in a Common Data Model

The study protocol indicated that all databases were standardized to the Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM), version 5.3 [16]. The OMOP CDM includes a standard representation of healthcare experiences and common vocabularies for coding clinical concepts, which enables rapid and consistent analysis across varied data sources.

Consideration 3: Advanced Phenotyping

Planning an observational study for safety signal assessment must consider how to prioritize and validly define phenotypes of interest (e.g., adverse drug reactions) using administrative codes and algorithms in the chosen RWD source. As an opportunity for rapidity, the RSA-RWD Framework recommends that teams develop code lists and specify algorithms for commonly used phenotypes of interest in advance of conducting rapid RWD studies (i.e., “advanced phenotyping”).

Identified Enabler 1: Create or Utilize Existing Medical Definitions and Phenotype Libraries

The selected study’s protocol presented the code lists and algorithms used to define the target cohorts, comparator cohorts, covariates, and outcomes. It is unknown whether these definitions were pre-existing. Developing a library of definitions for various medical conditions allows for future rapid RWD analysis.

Identified Enabler 2: Implement RWD Data Source Catalogues or Dashboards and Code Lists for Phenotype Development

The selected study considered several data sources that were standardized to the OMOP CDM. Knowledge of the OMOP coding system and of the type of information available in each data source was required to create appropriate definitions for the medical conditions of interest. Having this information summarized in data source catalogs or dashboards can provide a foundation for rapidly developing phenotype definitions as needed for future RWD analyses.

Identified Enabler 3: Prioritize the Phenotypes that are More Commonly Investigated

The outcome of interest in the selected study, hip fracture, is a frequently observed health issue, especially in elderly populations. Such commonly observed clinically significant conditions could be prioritized in phenotype development. Archives of these prioritized phenotypes can become a rich resource of pre-existing phenotypes for accelerated RWD analysis.

Consideration 4: Analytical Preparedness

Designing a high-quality RWD analysis plan for a safety signal assessment requires several decisions to be made related to patient characterization, comparator cohort definitions, measurement of exposure time, methods to control for confounding, and outcome assessment. As an opportunity for rapidity, the RSA-RWD Framework recommends that the creation or use of standard programming code, packages, and output templates can facilitate rapid results generation.

Identified Enabler 1: Pre-develop Standard Programming Codes

The selected study used a publicly available standard programming code, which enhanced efficiency and reduced potential for errors.

Identified Enabler 2: Establish Analytical Programs, Platforms, or Methodologies for RWD Analysis with Pre-specified Modules

The selected study used a pre-developed analytical platform equipped with methodologies tailored for RWD analysis (e.g., propensity scores, visualization tools).

Identified Enabler 3: Pre-develop Standard Output Templates

The protocol of the selected study described how the output will be organized into tables and figures however, specific output templates are not presented. Given that the described tables are common to RWD analyses (e.g., attrition, descriptive characteristics, covariate balance, event counts, incidence rates), developing standard templates for future analyses is an opportunity for rapidity.

Consideration 5: Conceptual to Operational Question Mapping

The ICH M14 Guideline [5] outlines the differences between conceptual and operational definitions for exposures, outcomes, and confounders. The relative strengths, limitations, and uncertainties in applying an operational definition to closely approximate a conceptual definition and the need for validation are important considerations for all RWD analyses, whether rapidly executed or not. However, since rapid analyses will generally afford less opportunity for exhaustive consideration of these important topics, it is recommended that teams apply attention to the following identified enablers.

Identified Enabler 1: Validity of Data Collected in a Real-world Setting

The selected study analyzed RWD from administrative healthcare databases, which are collected during routine healthcare delivery rather than for research purposes. No standardized methodology was implemented to validate the recorded information. Discrepancies may exist between database records compared to true medical conditions or drug exposures (e.g., drug dispensing does not necessarily mean the medication was consumed). In the selected RWD study, because codeine-containing products can be purchased over the counter in the UK, drug exposure captured in the Clinical Practice Research Datalink (CPRD) database may not represent true exposure. In contrast, since these medications require prescription in the US, drug exposure captured in the US claims databases is more likely to be accurate and complete.

Identified Enabler 2: Operational Definition and Communication of Validation Efforts

The objective of the selected study was to assess whether exposure to tramadol, relative to codeine, causes a different risk of experiencing hip fracture in one year [7]. To achieve that goal, the investigators developed multiple operational definitions for the target cohort, comparator cohort, and outcome. The investigators not only replicated the operational definitions from a previously published study (Wei, et al., 2020) [8] but also used a modified approach to better represent conceptual definitions. The protocol specified all operational definitions and provided performance characteristics from validation of the outcome definitions. There are notable differences between the conceptual definitions in the original medical question and the operational definitions used in the two RWD studies (Wei, et al. and Voss, et al.) [7, 8], which have led to different results.

Identified Enabler 3: Translation of Evidence Generated from Operational Definitions

To allow translation of evidence generated from RWD analyses that have operationalized conceptual definitions, careful control for bias is essential. In the selected study, multiple analytical strategies were applied to improve comparability of the target and comparator cohorts and detect potential confounding in the risk estimate, including additional exclusion criteria, propensity scores matching, and negative control outcomes. Though residual bias cannot be completely ruled out, these measures significantly improved validity. Importantly, the investigators explicitly described limitations to their approach (e.g., new use definition) and unexpected findings (e.g., limited importance of morphine milligram equivalents conversion), which provides readers with appropriate context for interpreting the results.