“Real-world evidence” (RWE) has been defined as the information derived from analysis of routinely collected real-world data (RWD) relating to a patient’s health status or the delivery of health care from a variety of sources other than traditional clinical trials.1 The use of RWE to support regulatory decision making is not new. For decades, such data have been used in the postauthorization phase for safety signal evaluation, risk management, and for studies to support life cycle benefit-risk evaluation. A review of postmarketing assessments conducted by the European Medicines Agency (EMA) in 2019, showed that noninterventional studies commonly contributed to the evaluation of referrals related to both products’ safety and efficacy.2 Although randomized clinical trials (RCTs) represent the gold standard for studying drug efficacy because they prevent systematic bias in allocation of treatment,3 they cannot answer certain questions, for example, effectiveness under normal conditions of use, and may not be practical in some circumstances, for example, in very rare diseases or populations. The rapid pace of change in the scientific and technological landscapes is shifting the regulatory landscape. An increasing number of medicines, such as advanced therapy medicinal products (ATMPs) and orphan products for conditions with significant unmet need, face challenges when aligning with the traditional drug development pathway, where traditional RCTs may be unfeasible, unethical, or less well suited to “precision medicines” that increasingly require analysis on subsets of patients on complex treatment pathways.1, 4, 5

Whereas methodological challenges remain before RWE can become a routine part of decision making across all parts of drug development,6 RWE can still have a substantial impact on regulatory decision making, for example, by informing on the natural history of disease and standards of care, by contextualizing results of uncontrolled trials when used as comparator groups of patients for single arm trials, or by collecting follow-up data to generate postauthorization evidence on long-term safety and effectiveness of medicinal products.1

Currently, there is a lack of detailed information on the use of RWE in marketing authorization applications in terms of objectives (for example, to assess safety or efficacy endpoints, use in special populations, or inform risk management planning), data sources, methods, the strengths and weaknesses of these, and the outcome of its assessment. Previously published studies on the topic have been dependent on information available in the public domain, for example, European Public Assessment Reports (EPARs), and have therefore based their evaluation on products that complete the authorization process rather than those submitted for authorization,2, 4, 7 thereby ignoring evidence submitted in withdrawn applications. In addition, considering exclusively the evidence leading to a final opinion by the EMA Committee for Medicinal Products for Human Use (CHMP) provides only a partial picture of the usefulness of RWE for marketing authorization. An evaluation of the totality of the information, including the nonpublished assessment of products that were withdrawn or for which additional data were requested during the assessment procedure, would support the provision of recommendations for best practice, improve the efficient use of RWE for regulatory applications, and enable assessment of the needs for data validation, expertise, and training. As recently pointed out, clarity is needed around terminology and practice when considering interventional or noninterventional designs, primary collection or secondary use of data, and characteristics of comparison group(s), as well as an assessment of prognostic determinism for the corresponding cause-effect association.8

This article describes the first phase of a project aiming to evaluate the impact of RWE in the decision-making process of medicinal products. The ultimate objective is to develop guidance targeted at various stakeholders to promote use of high quality RWD in regulatory decision making.9 It provides a description of the characteristics and subsequent approval status of RWE included in new marketing authorization applications (MAAs) and extensions of indication (EOIs) for previously authorized products submitted to the EMA in 2018 and 2019.

METHODS Definitions

RWD were defined as routinely collected patient-level data relating to their health status and/or the delivery of health care from a variety of sources other than RCTs. RWE was then defined as the information derived from RWD.1 For the purpose of this study, these conceptual definitions were translated into operational criteria, as illustrated in Table 1. In the context of clinical trials, the definition includes the use of observational data to support and complement RCTs, even types of clinical trial considered interventional.

Table 1. Types of real-world data / real-world evidence included/excluded in new MAAs and EOIs for already authorized products Included as RWD/RWE Not included as RWD/RWE Patient-based non-interventional pre- or postauthorizsation studies performed to support the marketing authorization application (primary and/or secondary use of data) Use of RWD source(s) (e.g., registry, electronic health care records, medical charts, etc.) to provide comparator (historical or contemporaneous) arms in single arm trials Use of RWD source(s) in the context of RCTs (e.g., to assess the representativeness of the control arm) Use of RWD data source(s) to support the implementation of clinical trials (e.g., by facilitating identification of eligible patients or type of data collection) Product-related literature review (e.g., on real-world safety data of the product in other indications) Non-product related literature review (for example reviews related to the natural history of the targeted disease or comorbidities associated with the disease) Use of aggregated epidemiological data from multiple sources where attribution is unclear Interventional studies (phase I, II), including preclinical studies, toxicological studies, dose-response studies, drug-drug interaction studies Phase III/IV studies without RWD/RWE use (e.g., single-arm study without comparator group from RWD) Open-label follow-up studies of clinical trial patients Routine pharmacovigilance activities Active surveillance based on spontaneous reporting Surveys not based on individual patients (e.g., surveys of physicians to assess awareness of risk minimization measures) EOIs, extensions of indications; MAAs, authorization applications; RCT, randomized clinical trial; RWD, real-world data; RWE, real-world evidence. Products of interest

All applications for MAAs and EOIs submitted from January 1, 2018, to December 31, 2019, and assessed by the CHMP were included. This study period was used as it was anticipated that CHMP assessment reports (ARs) would have been finalized by the time of the study lock point for data extraction of August 12, 2020. Applications for generic medicines (medicines developed to be the same as an already authorized reference medicine), informed consent applications (referring to the pharmaceutical, preclinical, and clinical data of a reference product), and well-established use applications (based on results from the scientific literature when the active ingredient has been used for more than 10 years and its efficacy and safety have been well-established) were excluded to avoid either a double-counting of the same information or use of information submitted outside the study period.

Data extraction and analysis

New MAAs and EOIs were identified from the EMA’s product information and application tracking system (SIAMED)10 and exported into an Excel file with administrative information: this list was inclusive of the European Commission Register of medicinal products (the Community Register) of marketing authorization of the products in the European Union,11 and so was considered a complete list of eligible products. Information on pediatric use, ATMP status, and PRIME status (“priority medicines” status that enhances EMA’s support for the development of medicines that target an unmet medical need) were collected from internal data sources.12, 13 The presence of RWD or RWE in the application was primarily identified from the CHMP AR retrieved from electronic archives held at the EMA. The AR and the version of the Risk Management Plan (RMP) most closely linked to the authorization of the product were manually searched for a series of characteristics of RWD/RWE. For MAAs and EOIs withdrawn during the application procedure, the last AR available was searched. Where the information available in the AR or RMP was unclear or did not contain enough details, additional files submitted by an applicant as part of the dossier were consulted. MAAs and EOIs were assigned randomly to seven reviewers who performed the screening and extraction of the required data by completing an online survey based on a predefined list of variables and questions that were to be abstracted from the available product information. Two electronic files—one for the initial MAAs and one for the EOIs—were compiled and made available for statistical analysis. The analysis of RWD/RWE consisted of a descriptive analysis of the information collected, the number and percentages of MAAs/EOIs for which RWD/RWE was submitted, and the distribution of RWD/RWE submission by: Anatomical Therapeutic Chemical (ATC) classification; whether pre- or postauthorization; whether included as a main or supportive study (“main” referring to studies identified as those pivotal to the assessment process, typically identified in a specific section of the assessment report)14; RMP category14; the objective studied (safety, efficacy, disease epidemiology, drug utilization, abuse of drug, and other), data sources used (electronic health records, claims data, registries as identified in the AR, data from compassionate use program, spontaneous report database, reuse of data from observational studies, linked data sources, and other), and study design (cohort, case-control, cross-sectional, ecological, case-only design, and other).

Quality control

The strategy for identifying and categorizing the information was piloted with seven investigators independently extracting and discussing the data for an initial three MAAs to ensure common understanding of how to meet the study’s objectives and to agree on the criteria for the categories of information to be included and excluded. During the actual review stage, regular meetings were organized among the reviewers to discuss data where an investigator was uncertain about their status as RWD/RWE and to agree on an outcome by consensus. Two additional investigators acting as quality controllers were appointed to cross-check a random sample of 33 MAAs and 37 EOIs (error margin of 10%; confidence level of 80%) with stratification at reviewer level. In case of disagreement, the reviewer provided written comments explaining their decision and initiated a discussion with the investigator to reach consensus.

RESULTS

From January 1, 2018, to December 31, 2019, the EMA received 201 MAAs and 163 EOIs applications, of which 158 and 153, respectively, were within the scope of the study after excluding applications relating to generic products (n = 38), informed consent (n = 14), and well-established use applications (n = 1). Of the remaining eligible products, 63 MAAs (39.9%) and 28 EOIs (18.3%) contained references to RWD/RWE. The distribution by the ATC classification system is shown in Table 2.

Table 2. Use of RWD and RWE by ATC classification ATC classification

Initial MAAs

n with RWE / total assessed (%)

EOIs

n with RWE / total assessed (%)

A Alimentary tract and metabolism 6/13 (46.2) 3/14 (21.4) B Blood and blood forming organs 7/11 (63.6) 7/10 (70.0) C Cardiovascular system 0/4 (0) 0/3 (0) D Dermatologicals 0/1 (0) 0/3 (0) G Genito urinary system and sex hormones 0/0 (0) 0/0 (0) H Systemic hormonal preparations, excl. sex hormones, and insulins 1/8 (12.5) 0/0 (0) J Anti-infectives for systemic use 14/25 (56.0) 1/19 (5.3) L Antineoplastic and immunomodulating agents 23/61 (37.7) 12/78 (15.4) M Musculo-skeletal system 1/1 (100) 2/2 (100) N Nervous system 8/17 (47.1) 1/5 (20.0) P Antiparasitic products, insecticides and repellents 0/0 (0) 0/0 (0) R Respiratory system 1/8 (12.5) 1/14 (7.1) S Sensory organs 2/6 (33.3) 0/2 (0) V Various 0/3 (0) 1/3 (33.3) ATC, Anatomical Therapeutic Chemical; EOIs, extension of indication; MAAs, marketing authorization application; RWD, real-world data; RWE, real-world evidence.

At the time of data extraction from SIAMED on August 12, 2020, 73 MAAs had been authorized (37 with RWE), 26 had been withdrawn or refused (4 with RWE), and 59 were still under evaluation by the EMA (20 with RWE). Forty-eight had a pediatric indication (22 with RWE), 2 had PRIME status (neither with RWE) and 5 were ATMPs (3 with RWE). For EOIs applications, 104 had been authorized (19 with RWE), 8 had been withdrawn or refused (1 with RWE), and 41 were still under evaluation (8 with RWE). Fifty-eight had a pediatric indication (15 with RWE), 6 had PRIME status (4 with RWE) and none were considered ATMPs.

Characteristics of the RWE that were used in support of initial MAAs and EOIs are shown in Table 3. The 63 MAAs included a total of 117 studies across products. RWD was submitted within the application as evidence from the pre-authorization phase for approximately one third of the products and had an exclusively supportive role in the benefit-risk assessment in 75.0% of cases. Applications including RWE during the postauthorization phase consisted mainly of RWE-based category 3 (required) studies of the RMP (74.1%). RWD sources were most commonly registries (60.3%), followed by hospital data (31.7%), and were mainly included to support safety (87.3%) and efficacy (49.2%) objectives. The most frequently used registries were disease registries (33.3%) and product registries (those where the registry used was focused on patients receiving the specific medicinal product of interest; 14.3%). Cohort studies were the most frequently used observational design (88.9% of products).

Table 3. Characteristics of RWD/RWE found in initial MAAs and EOIs Characteristics of RWD/RWE studies used

Initial MAAs

n (%)

Total MAAs = 63

EOIs

n (%)

Total EOIs = 28

Number of RWE studies included in applications Total 117 36 Number of RWE studies per application 1 29/63 (46.0) 23/28 (75.0) 2 22/63 (34.9) 3/28 (10.7) ≥ 3 12/63 (19.1) 2/28 (14.3) Time of implementation of RWE studies in applications Pre-authorization 9/63 (14.3) 13/28 (46.4) Postauthorization 43/63 (68.3) 12/28 (42.8) Pre-authorization and postauthorization 11/63 (17.4) 3/28 (10.7) Whether RWE studies to support pre-authorization were included as main or supportive studies or a combination of both main and supportive Main study(ies) 3/20 (15.0) 4/16 (25.0) Supportive study(ies) 15/20 (75.0) 12/16 (75.0) Both main and supportive stud(ies) 2/20 (10.0) 0/16 (0.0) EU RMP category for at least one postauthorization study, if requested Category 1 (imposed as condition of MA) 11/54 (20.3) 1/15 (6.6) Category 2 (specific obligation of MA) 3/54 (5.6) 0/15 (0.0) Category 3 (required) 40/54 (74.1) 14/15 (93.3) Objective of RWE studies Safety 55/63 (87.3) 23/28 (82.1) Efficacy 31/63 (49.2) 15/28 (53.6) Disease epidemiology 5/63 (7.9) 3/28 (10.7) Drug utilization 13/63 (20.6) 6/28 (21.4) Abuse of drug 6/63 (9.5) 0/28 (0.0) Other objectives 8/63 (12.6) 3/28 (10.7) Data sources Electronic health care records from primary care 8/63 (12.7) 2/28 (7.1) Electronic health care records from secondary care 8/63 (12.7) 0/28 (0.0) Medical records from primary carea 8/63 (12.7) 5/28 (17.9) Hospital data 20/63 (31.7) 7/28 (27.0) Claims data 5/63 (7.9) 2/28 (7.1) Prescription data 6/63 (9.5) 3/28 (10.7) Dispensing data 5/63 (7.9) 1/28 (3.6) All registriesb 38/63 (60.3) 13/28 (46.4) Disease registry 21/63 (33.3) 9/28 (32.1) Product registry 9/63 (14.3) 3/28 (10.7) Other registriesc 13/63 (20.6) 2/28 (7.1) Data from compassionate use program 2/63 (3.2) 1/28 (3.5) Spontaneous reportsd 4/63 (6.3) 3/28 (10.7) Re-use of data from observational studies 4/63 (6.3) 1/28 (3.6) Linked data sources 3/63 (4.7) 1/28 (3.6) Other data sourcese 18/63 (28.6) 5/28 (17.9) Design of noninterventional studies, if used as RWE Cohort studies 56/63 (88.9) 22/28 (87.6) Case-control 2/63 (3.1) 0/28 (0.0) Cross-sectional 3/63 (4.7) 2/28 (7.1) Otherf 8/63 (12.6) 3/28 (10.7) EOIs, extensions of indication; EU RMP, European Union Risk Management Plan; MA, marketing authorization; MAAs, marketing authorization applications; RWD, real-world data; RWE, real-world evidence.

The 28 EOIs products included a total of 36 studies in their applications. These studies were mainly conducted prior to the EOIs (57.1%) and were supportive (75.0%) rather than main studies. Applications foreseeing use of RWE during the postauthorization phase consisted mainly of RMP category 3 studies (93.3%). RWD sources were mainly registries (46.4%) and hospital data (27.0%), and were typically used to support safety (82.1%) and efficacy (53.6%) objectives. The most frequently used registries were disease registries (32.1%) and products registries (10.7%). Cohort studies were the most frequently used design (87.6% of products).

We analyzed whether the objectives of the RWE generation differed according to whether the studies were submitted pre-authorization or were proposed as part of the postauthorization phase. Of 36 MAAs and EOIs supported with pre-authorization RWE, 27 (75.0%) were directed at safety and 27 (75.0%) were directed as efficacy/effectiveness. Of 69 MAAs and EOIs with postauthorization RWE, 65 (94.2%) were directed at safety and 30 (43.5%) were directed as efficacy/effectiveness.

The purpose for which the RWE was collected varied but included: external/historical comparators, identification of patients for noninterventional studies, data collection on safety endpoints, data collection on efficacy, data collection on disease epidemiology, measuring effectiveness of risk minimization measures, comparison of surrogate and clinical endpoints, studying patterns of drug utilization, and feasibility analyses. Where stated, the most common setting of the RWE studies was the European Union (including the United Kingdom), followed by the United States and the rest of the world.

Thirty-eight MAAs and 13 EOIs made used of one or more registries a part of their application (Table 3, Figure 1). For initial marketing authorization applications, these were more frequently postauthorization studies (30 postauthorization and 11 pre-authorization), whereas for the EOIs applications, these were more balanced (7 pre-authorization and 9 postauthorization). We found that registry studies were imposed either as category 1 (imposed as condition to the terms of marketing authorization) or category 2 (specific obligation to the terms of marketing authorization) in the EU RMP, in 14 of 158 MAAs (8.9%) and 1 of 153 EOIs (0.7%), with 4 coming from product registries and 5 from disease registries.

Pie charts showing the distribution of registry types amongst initial Marketing Authorization Applications (n = 38) and Extensions of Indication (n = 13).

Quality control

Of the 33 MAAs and 37 EOIs cross-checked, 15 and 7 applications, respectively, resulted in disagreement. The nature of the disagreement related only to the detail of the data collected, for example, relating to pediatric indication status, the type and origin of RWD sources, and uncertainty around the study design in the documentation. For some investigators, the discrepancies between reviewer and investigator, and the discussion to resolve them required the investigator to repeat the data extraction.

DISCUSSION

This study examined when and how RWE was used to support MAA for new products and EOIs for products currently marketed. We found that RWE was used in 39.9% of MAAs (mainly in the postauthorization setting) and 18.3% of EOIs (balanced between pre- and postauthorization) submitted to the EMA. It is perhaps surprising that RWE was less used in support of EOIs applications considering RWE would already have been generated in the postmarketing setting. It could be that the RWE submitted for EOIs applications was of greater importance in the decision-making process, whereas perhaps being more descriptive/supportive for MAAs—we found that 15% of MAAs and 25% of EOIs had RWE present as a main study. Alternatively, for EOIs, it could be that concerns around safety are already addressed by ongoing pharmacovigilance activities following the initial marketing authorization, so there is less of a role for additional RWE, which historically has principally been used for establishing safety. Such considerations will be the topic of further research.

When used pre-authorization, supportive studies commonly evaluated efficacy and effectiveness, whereas postauthorization studies were mainly RMP category 3 (for studies included in RMPs) focused on safety. The main RWD sources used for both pre- and postauthorization studies were disease registries, hospital data, claims/prescriptions/dispensing data, and electronic health care records.

Antineoplastic and immunomodulating agents accounted for over a third of initial MAAs and a half of EOIs submissions, and as such many applications where RWE was used. Many of these products are indicated for rare—often fatal diseases—with applications being based on uncontrolled clinical trials instead of traditional RCTs. In such cases, the use of RWE in submissions is to support demonstration of efficacy, for example, through provision of external comparators in single arm studies. Depending on the role of the external comparator, the contribution could be designated as a “supportive” study or as part of the “main” study. Alternatively, the life-threatening nature of many cancers means that new treatments are often authorized under accelerated procedures and might be granted a conditional marketing authorization, pending additional evidence submitted postapproval. In this context, RWE can support postauthorization follow-up to confirm the long-term efficacy and safety needed for a full MAA, and in this context could be considered a “supportive” study. Work is ongoing to detail the type, purpose, and influence of RWD/RWE in the marketing authorization process.

A predecessor to this study by Bouvy et al. found that for 392 products that received a positive CHMP opinion from 2005–2013, there were 31 registries that were requested for 30 products (7.7%) in total (65% were product registries, 35% were disease registries) with 71% having a primary safety objective.15, 16 This frequent use of product registries led to the establishment of the EMA Patient Registry initiative supporting the use of existing patient registries for the postauthorization benefit–risk monitoring of medicinal products.17 This was subsequently followed up with proposals for operational methods for increasing the use of patient registries in medicines regulation by addressing: the nature of the data collected and registry quality assurance processes; registry governance, informed consent, data protection, and sharing; and stakeholder communication and planning of benefit-risk assessments.18 Although following a different methodology, we found an apparent increase in the extent of registry use compared to that found by Bouvy et al., with 24.1% of all MAA and 8.5% of all EOIs applications making use of registries. We found registries were used in 56.0% of cases where RWD was used: in 33.0% of applications these were disease registries, 13.2% product registries, and 16.5% other registries.

Previous studies have also highlighted the increasing importance of RWE to the regulatory decision-making process. A review of 52 referrals completed by the EMA between January 2013 and June 2017, showed that noninterventional studies contributed to the evaluation of 59.6% of the referrals related to product safety and of 34.6% related to product efficacy.2 A further review of new drug applications identified 73 registries planned postauthorization, of which 39 aimed to collect safety outcomes and 7 aimed to collect safety outcomes and real-world effectiveness data.7 Using publicly available regulatory documents, a review of 415 EMA-approval decisions on pharmaceuticals from January 1, 1999, to May 8, 2014, found that 44 indications for 35 products were approved on the basis of uncontrolled study data, including 8 extensions of indication for treatments with RCTs in other indications and 36 indications for products in which there were no RCT results in an approved indication.4 This study did not address the possible use of RWE as alternative supporting evidence. In the field of orphan medicinal products, a review of 125 dossiers published between 1999 and 2014 found that 12% did not include evidence from clinical trials but were based on literature reports, observational studies, or compassionate use programs.19

The use of RWE in decision making for marketing authorizations depends on many factors, as summarized by Cave et al. in three categories: operational, technical, and methodological.1 Essential factors to consider include: the information available in the data source, the quality and validity of this information, the design of the data collection, the statistical analytical plan supporting data analysis and interpretation, and the likelihood of bias due to the unblinded, uncontrolled, or nonrandomized treatment allocation.20 In some circumstances, scientific evaluation of the efficacy and safety of medicines prior to granting a marketing authorization may be supported by observational evidence where an RCT is deemed not feasible or unethical. Prior consultation with the EMA via the procedure for scientific advice and protocol assistance is generally recommended to agree on the adequacy of a source of RWE to support marketing authorization.17

Our study has a number of strengths. First, we focused on MAAs and EOIs submitted over a 2-year period rather than those authorized, as performed by other recent studies.2, 4, 7 This allowed an exhaustive evaluation of the characteristics of RWE, including those in applications withdrawn during the evaluation procedure. Further research is needed to assess the impact of some of these characteristics on the acceptability of the authorization, and reasons for decisions on withdrawal and refusal decisions. Second, the investigators had access to information available at the EMA and the original application files submitted by marketing authorization applicants, some of which would not be available in the public domain. It allowed us to further validate or complement the information available in published EPARs and RMPs. Third, to minimize heterogeneity in the way RWE was described and classified across applications, a quality control mechanism was put in place to take into account the fact that data extraction had to be performed by screening electronic documents with a distribution of the workload among seven investigators. This quality control mechanism did not identify significant systematic differences between investigators and contributed to consistent data extraction and categorization. Finally, the exhaustive repository of RWE submitted to EMA over a 2-year period will allow further, more detailed evaluations of the usefulness and acceptability of RWE in application procedures.

The main limitation is the lack of internationally agreed guidance for the operational definitions of RWD and RWE detailing which type of data should allow to identify an application containing such features. The inclusion/exclusion criteria presented in Table 1 are the result of a consensus between the authors considering unforeseen types of data emerging during the period of data extraction and were guided by a shared view of RWD as those having high external validity and arising from routinely collected patient level data. This was informed by a definition of RWD excluding traditional clinical trials1—similar to one used previously by the FDA21—although we included trials that involved use of data collected from other sources, for example, single arm trials using such data as a comparator. However, we acknowledge that the terminology around this topic is nuanced: a more recent FDA definition of RWE includes evidence generated by different study designs or analyses, such as randomized and pragmatic trials.22 For the purposes of this study, we took the view that traditional RCTs and open-label follow-up of these RCTs do not constitute RWD/RWE, as the data generated in the open-label follow-up would not be independent from the active phase of the RCT and therefore not necessarily represent a new research question addressed by RWD/RWE. We also considered the population of the original trial would typically not be representative of the underlying patient population. This is because trial design frequently lacks certain types of patients, a phenomenon partly mediated through the exclusion criteria inherent to the trial (for example, vulnerable patients, those with multiple comorbidities, or who are pregnant) and partly because trials routinely under-recruit certain categories of subjects, for example, ethnic minorities or those from socioeconomically disadvantaged backgrounds. Although we included patient surveys, other surveys, for example, healthcare professionals’ awareness of risk minimization measures, were excluded: the focus of much of the RWD was on patient-level data used to address drug safety or effectiveness outcomes, and this generally was not directly addressed by surveys of healthcare professionals.

Arguably, decisions around what was considered real-world were based on criteria defined by the investigators and other research groups might have classified studies differently. Given that definitions of what constitutes RWD and RWE overlap with traditionally used dichotomous classification of clinical research as “interventional vs. noninterventional,” “prospective vs. retrospective,” “externally valid vs. selected populations,” and “primary vs. secondary data collection,”8 there is undoubtedly a need for internationally agreed criteria in this regard. A second limitation is that not all procedures had been finalized by August 12, 2020, which explains that, for 32.1% of products investigated, there was no information available on the final status of the application. The proportion of product with applications that are refused or withdrawn might change with time.

Future work will describe in further detail those studies that contributed RWE to the regulatory application, for example, describing requests for, and outcome of, scientific advice, and focusing on how important RWD and RWE are to the regulatory evaluation and decision-making process. Such characteristics are more qualitative in nature and were beyond the scope of the current quantitative study but will be analyzed in more detail in a second phase of this project. It is anticipated that this research will contribute to generating the evidence-base required to drive the EMA’s and other regulators’ use of RWE in decision making, ultimately advancing patient-centered access to medicines in partnership with healthcare systems.9

ACKNOWLEDGMENTS

The findings of this study have been presented to, and comments gratefully received from, the following EMA committees: the Committee for Medicinal Products for Human Use (CHMP), the Pharmacovigilance Risk Assessment Committee (PRAC), the Scientific Advice Working Party (SAWP), the joint HMA/EMA Big Data Steering Group, and the PRAC interest groups.

FUNDING

No funding was received for this work.

CONFLICT OF INTEREST

The authors declared no competing interests for this work.

AUTHOR CONTRIBUTIONS

All authors contributed to the design of the study, performed the research and contributed to the drafting of the manuscript. K.P. coordinated the implementation of the study. C.Q. and K.P. compiled and analyzed the data and undertook the quality control procedures. X.K. initiated the study. The 158 MAAs were assigned randomly to 6 reviewers (R.F. = 11, V.S. = 31, R.D. = 31,