1 INTRODUCTION

Integration of new research evidence into care delivery is often slow and inefficient. As a result, patients do not consistently receive high-quality, evidence-based care. Adopting the tenets of a Learning Health System (LHS) enables organizations to transform care delivery, improve patient health and experience, accelerate generation of patient-centered knowledge, and reduce costs by decreasing the time for providers and health systems to ascertain and act on research evidence.1

An LHS is a system in which internal data and experience are systematically integrated with external evidence, and that knowledge is seamlessly put into practice.2 Many organizations have explored and refined facets of an LHS, defining LHS researcher core competencies3 and core values,4, 5 articulating LHS priorities as the future of health services research,6 and exploring how to transform an organization into an LHS.7 Others have proposed frameworks for ethics8 and values to guide data collection and patient engagement in an LHS8 from a care delivery perspective.4, 8, 9 Methodologists have developed pragmatic clinical trials and robust, rapid-cycle, non-randomized methods for embedded research.10 Approaches to harmonize information technology infrastructure to facilitate access to and use of health systems data for learning are emerging across multiple settings.11 Publications describing LHSs reveal a complementary relationship between embedded pragmatic research and quality improvement (QI) activities.12

Despite this progress, knowledge about developing, operationalizing, and evaluating the impact of LHS initiatives is scarce. While numerous papers list the desired constructs (fundamental components) of an LHS, details about how to operationalize those constructs, maximize the relationships among them, and assess their effectiveness and impact are lacking.13-15 A systematic review of LHS programs concluded that most work has been conceptual and unapplied, with “minimal focus on evaluating the impact.”16 Demonstrating the impact of an LHS could be a catalyst to higher acceptance and adoption of LHS concepts by health systems worldwide.

Building on decades of research partnership with care delivery, Kaiser Permanente Washington (KPWA) launched its LHS program in June 2017 as part of a strategic initiative to rapidly improve care and services for patients by integrating research capabilities such as evidence reviews, advanced analytics, program design and evaluation, and implementation science into strategic decision making. Group Health Cooperative was an early adopter in operationalizing the LHS and was acquired by KPWA in 2017.17 We created the LHS program to operationalize the LHS concept in our system, which is a single institution where the organization has control over many of the components of the LHS. This is distinct from broader LHS definitions that may work across multiple settings and institutions. The KPWA LHS program is led by an interdisciplinary team of operational, research, and medical staff at a research institute with 62 faculty and 246 staff. To address the lack of operational LHS models and to evaluate impact, KPWA LHS leadership reviewed published LHS conceptual models and measurement approaches. To achieve an evaluable depiction of an LHS, we constructed a logic model capturing the relationship between program resources, activities, and intended outcomes.

Here we present the KPWA LHS Logic Model, which provides a broad list of constructs relevant to LHS programs, depicts their relationship to LHS operations, harmonizes terms across models, and offers measurable operationalizations of each construct to guide other LHSs. The model identifies essential LHS inputs, provides transparency into LHS activities, and defines key outcomes to evaluate LHS processes and impact. Since the COVID -19 pandemic provided a unique opportunity to explore how rapid translation of science into practice occurs, we provide reflections on the most helpful components of the model and identify areas that need further improvement using examples from deployment of the KPWA LHS model during COVID-19 to illustrate. Our purpose is to help organizations that want to establish an LHS program develop a roadmap for their organization, understand how LHS constructs relate to one another when operationalized in practice, and evaluate and improve their progress.

2 METHODS

We convened an interdisciplinary working group of LHS researchers with expertise in implementation science, QI, operations, communications, and translational research to identify core LHS constructs. We used a narrative review approach, which is appropriate when the question of interest is too broad or the body of evidence too sparse for a systematic review.18 One team member (KM) conducted a literature search to identify peer reviewed articles that included LHS models or listed LHS core constructs. We searched the PubMed and Embase databases, which have broad coverage of the academic literature. Search terms included variations of the base “learning health system,” including learning health system framework, learning health system model, learning health system program, and closely related synonyms. We did not include the term “translational research” or other related synonyms because these additional terms expanded the focus of the review beyond our intended scope. We also searched for common construct terms from existing LHS models (eg, “patient engagement,” “leadership”) and synonyms combined with the above terms and synonyms.

We scanned titles and abstracts of identified articles and read methods and background sections of studies that measured or attempted to measure LHS constructs. Since the field is new, we found widespread inconsistency in terminology. Therefore, we identified additional literature by following lead authors frequently cited in relevant articles. Ultimately, 17 articles met criteria for further review (Table 1).

TABLE 1. Citation list of models and frameworks analyzed to identify LHS constructs (listed chronologically) Model or framework Citation 1 Physiology of a Learning System Model Bohmer R. Designing care: aligning the nature and management of health care. Harvard Business Review Press 2009. 2 Rapid-Learning Health Care System Model Greene SM, Reid RJ, Larson EB. Implementing the learning health system: from concept to action. Ann Intern Med. 2012;157(3):207-210 3 Requirements for a Learning Health System Friedman C, Rigby M. Conceptualising and creating a global learning health system. Int J Med Inform. 2013;82(4):e63-71. 4 Characteristics of a Continuously Learning Health Care System McGinnis JM, Stuckhardt L, Saunders R, Smith M. Best care at lower cost: the path to continuously learning health care in America. National Academies Press; 2013. 5 Framework for a National-Scale LHS Bernstein JA, Friedman C, Jacobson P, Rubin JC. Ensuring public health's future in a national-scale learning health system. Am J Prev Med. 2015;48(4):480-487. 6 LHS Research Challenges and Questions Friedman C, Rubin J, Brown J, et al. Toward a science of learning systems: a research agenda for the high-functioning Learning Health System. J Am Med Inform Assoc. 2015;22(1):43-50. 7 LHS-related IOM Reports IOM Roundtable on Value & Science Driven Care. Integrating Research and Practice: Health System Leaders Working Toward High-Value Care: Workshop Summary. National Academies Press; 2015. 8 Learning Health Care System Framework Psek WA, Stametz RA, Bailey-Davis LD, et al. Operationalizing the learning health care system in an integrated delivery system. EGEMS (Wash DC). 2015;3(1):1122. 9 Patient-Centered Rapid Learning System Model Wysham, N. G., Howie, L., Patel, K., Cameron, C. B., Samsa, G. P., Roe, L., … & Zaas, A. (2016). Development and Refinement of a Learning Health Systems Training Program. eGEMs, 4(1). 10 Learning Health Care System in the Veterans Health Administration Atkins D, Kilbourne AM, Shulkin D. Moving From Discovery to System-Wide Change: The Role of Research in a Learning Health Care System: Experience from Three Decades of Health Systems Research in the Veterans Health Administration. Annu Rev Public Health. 2017;38:467-487. 11 Learning Health System Consensus Core Values Friedman CP, Rubin JC, Sullivan KJ. Toward an Information Infrastructure for Global Health Improvement. Yearb Med Inform. 2017;26(1):16-23. 12 Learn from Every Patient Lowes LP, Noritz GH, Newmeyer A, et al. “Learn From Every Patient”: implementation and early results of a learning health system. Dev Med Child Neurol. 2017;59(2):183-191. 13 Framework for Local and External Evidence Integration Guise JM, Savitz LA, Friedman CP. Mind the Gap: Putting Evidence into Practice in the Era of Learning Health Systems. J Gen Intern Med. 2018;33(12):2237-2239 14 Heimdall Framework for Supporting Characterisation of Learning Health Systems McLachlan S, Potts HWW, Dube K, et al. The Heimdall Framework for Supporting Characterisation of Learning Health Systems. J Innov Health Inform. 2018;25(2):77-87. 15 Conceptual Framework for Value-Creating Learning Health Systems Menear M, Blanchette MA, Demers-Payette O, Roy D. A framework for value-creating learning health systems. Health Res Policy Syst. 2019;17(1):79. 16 Care and Learn Model Montori VM, Hargraves I, McNellis RJ, et al. The Care and Learn Model: a Practice and Research Model for Improving Healthcare Quality and Outcomes. J Gen Intern Med. 2019;34(1):154-158. 17 Multilevel Framework Harrison MI, Shortell SM. Multi-level analysis of the learning health system: Integrating contributions from research on organizations and implementation. Learning Health Systems.e10226.

The team reviewed the papers and identified constructs based on strength of conceptual or empirical support for impact on an LHS, consistency in definitions, alignment with our own experience and potential for measurement. Psek and colleagues' framework from Geisinger Health Systems of nine LHS operational components emerged as the most comprehensive and applicable starting point to define core constructs.7 We compared the components to our experiences and the literature and found important omissions and a need for clearer operational definitions for several components. We used other studies to identify additional constructs, compared them across models, and clearly defined each in measurable terms. We reviewed the list of constructs and definitions and came to consensus on those central to an LHS. Our goals were to reduce redundancy, align constructs with salient domains (while acknowledging overlap and some fluidity), and ensure a comprehensive list of constructs that are key to a high-functioning LHS. We combined constructs with different labels but overlapping definitions and parsed constructs that conflated underlying concepts.

We gathered examples of how each identified construct might be measured. We categorized each construct as a foundational LHS input, output or activity, or intended outcome. The resulting logic model portrays relationships among constructs and a framework to evaluate whether the LHS is achieving desired goals (Figure 1).

KPWA LHS Logic Model

3 RESULTS

The KPWA LHS Logic Model is based on the 17 frameworks and models identified in Table 1 and includes 24 constructs: 6 inputs, 9 outputs, and 9 outcomes. We describe each (Table 2) and provide examples of how to measure them to evaluate an LHS (Table 3).

TABLE 2. LHS logic model constructs with brief definitions Topic Description I. INPUTS A. People and partnership Personnel and relationships involved in establishing and maintaining learning activities within and external to the organization7 B. Health information infrastructure Integrated and interoperable system that supports the data requirements of multiple stakeholders, digitally captures the care experience and allows real-time access to knowledge for clinical care and learning7 C. Prioritization Process in which learning activities and opportunities are aligned with strategic goals across different levels of the organization7 D. Funding Mechanisms to fund the operational effort needed to enhance learning capability, as well as strategies for sustained funding of learning efforts7 E. Improvement infrastructure Leadership, policies and procedures to organize and facilitate improvement work19 F. Ethics and oversight Institutional guidance to navigate the differences, overlap, and similarities between quality improvement, clinical care, and research7 II. OUTPUTS A. Environmental scanning Internal and external assessment of the current state of an issue or practice to identify gaps and recommend best practices B. Evidence synthesis and translation Summarize the academic literature for a clinical or research question and explain the application of existing evidence to the issue at hand20 C. Data analytics Inspect, cleanse, transform, visualize, and model data with the goal of discovering useful information, informing conclusions, and supporting decision-making21 D. Design Design care based on evidence generated locally or elsewhere using pragmatic, timely, and flexible methods7, 17 E. Patient and family engagement Integrate stakeholder values, experiences, and perspectives into LHS projects22 F. Implementation support Facilitate the process of putting to use or integrating interventions in the care delivery setting22 G. Evaluation Collect data and analyze results to show what does and does not work17 H. Dissemination Share results to improve care17 I. Consultation The provision of expert advice and counseling to inform decision-making and promote learning23 III. OUTCOMES A. Knowledge-to-action Latency24 The average time lag for clinical practices to adopt research evidence to improve care for patients B. Systematic adoption of EBPs Evidence of actual performance of a practice in the system and target impacts of that performance in practice25 C. Systematic elimination of wasteful and ineffective practices Reduction in clinical and operational practices that are cost-ineffective or detrimental to health26 D. Population health Intermediate clinical health process and outcome measures for a population E. Care experience Patient satisfaction with care F. Utilization/Cost of care27 Utilization multiplied by the price of services, equipment, products, and prescription drugs G. Work life for care teams28, 29 Clinical care and research team experience H. Equity30 Fairness in processes, outcomes, and relative costs I. Programmatic return on investment The cost of the LHS program investment over the outcomes achieved in learning, health, experience, equity, work life of teams, and costs of care achieved across the projects the LHS program supports Abbreviations: EBP, evidence-based practice; LHS, learning health system. TABLE 3. Measurement of LHS logic model constructs Topic Level of analysis Available measurement Sample measures I. INPUTS A. People and partnerships Organization or setting Observation; Checklists

Do you have a team/department with dedicated time to meet to advance the program and solve problems?

Does your team include members with diverse skills including expertise in implementation, research translation, quality improvement, and operations?

Does your team have key stakeholder relationships in place to succeed?

B. Health information infrastructure Organization or setting Observation; Checklists

Does your organization have an EHR system?

To what extent is the system used to input data?

To what extent are there common data elements between information systems?

To what extent does the EHR and data output interface with external systems like national population health registries?

To what extent is population level data available at the point of care to care teams?

C. Prioritization Organization or setting Observation; Checklists

Is the LHS program directly cited in the organization's strategic plan? Indirectly?

Is there documented alignment between LHS priorities and the organization's operating plan?

D. Funding Organization or setting Administrative data

To what extent does the organization provide analytic time, personnel, and resources to support LHS activities?

To what extent is the LHS supported by federal, state, or local funding that is external to the organization?

E. Improvement infrastructure Organization or setting Administrative data To what extent does your organization have leadership, dedicated staff time, policies, and procedures in place to facilitate improvement efforts? F. Ethics and oversight Organization or setting Observation

To what extent do institutional guidelines and procedures exist to delineate quality improvement, clinical care and research?

Is there a regulatory body to oversee risk management for quality improvement, clinical care and research projects?

II. OUTPUTSa A. Environmental scanning Organization or setting Observation Count of environmental scans produced B. Evidence synthesis and translation Organization or setting Observation Count of rapid literature reviews produced C. Data analytics Organization or setting Observation

Count of reports that include original data analyses or descriptive data.

Count of data models created that care delivery adopts

D. Design Organization or setting Administrative data

Count of stakeholder convenings

Count of co-design sessions

Count of departments/providers/staff involved as partners

E. Patient and family engagement Organization or setting Administrative data

Count of patients and family members involved as partners

Count of projects that included patient and family member input

Documentation of how recommendations from stakeholders are applied

F. Implementation support Organization or setting Administrative data

Count of practice facilitator hours provided for care delivery initiatives

Count of projects that required implementation support

Count of resources and tools integrated into the EHR due to implementation support

Documentation of changes made to implementation process including integration of an implementation framework, changes to workflows, support for clinical decision-making, changes to implementation strategies, support pulling and applying data, and addressing context and barriers31

G. Evaluation Organization or setting Administrative data

Count of evaluation reports completed

Documentation of changes that occurred as a result of evaluation

H. Dissemination Organization or setting Observation

Count of internal and external publications, presentations, reports, and executive briefs

Count of partnerships established with external organizations

Documentation of LHS program's role in the internal spread of effective interventions

I. Consultation Organization or setting Administrative data Count of consultation requests completed, categorized by type of request III. OUTCOMES A. Knowledge-to-action Latency24 Organization or setting Observation; Administrative data

Average time from publication of high-quality evidence in academic literature to publication of an organizational guideline for a practice

Average time from release of an EBP guideline to uptake among a percentage of the organization's providers

B. Systematic adoption of EBPs Organization or setting Observation; Administrative data

Count of new EBPs adopted by the organization

Performance and impacts of existing EBPs at the organization

C. Systematic elimination of wasteful and ineffective practices Organization or setting Observation; Administrative data

Count of wasteful or ineffective practices reduced in the organization

Performance and impacts of reduction of wasteful and ineffective practices

D. Population health Individual consumer Qualitative or semi-structured interviews; Survey Sample measure sets include HEDIS, UDS, NCQA E. Care experience Individual consumer Qualitative or semi-structured interviews; Survey

Sample measure sets include CAHPS, Press Ganey

Patient retention

F. Utilization/Cost of care Individual consumer Administrative data Sample measures including utilization multiplied by the price of services, equipment, products, and prescription drugs G. Work life for care teams29, 32 Individual provider; Organization or setting Survey; Qualitative or semi-structured interviews; Focus-groups Sample quantitative measures including Maslach Burnout Inventory, internal provider and researcher satisfaction survey, Baldridge, Gallup Provider and staff retention H. Equity Organization or setting Administrative data

Count of projects measuring outcomes by race, ethnicity, language, age, and other socioeconomic factors

Count of projects using internal equity framework

I. Programmatic return on investment Organization or setting Administrative data

Cost of the LHS program investment over the outcomes achieved in learning, health, experience, equity, work life of teams, and costs of care

Diversity of funding sources

Abbreviations: EBP, evidence-based practice; EHR, electronic health record; HEDIS, Healthcare Effectiveness Data and Information Set; LHS, learning health system; NCQA, National Committee for Quality Assurance; Consumer Assessment of Healthcare Providers and Systems; UDS, Uniform Data System. a In addition to quantifying the number of deliverables for above categories, the quality of deliverables should also be assessed. 3.1 KPWA LHS logic model constructs 3.1.1 Inputs

Inputs are essential elements for an organization to successfully operate an LHS. Inputs tend to be manifest variables, meaning they can be directly measured or observed. An organization can measure the extent to which inputs are present to determine readiness to transform into an LHS.

People and partnerships

These are the personnel and relationships involved in establishing and maintaining learning activities within and external to the organization.7 This construct measures well-defined teams with diverse skillsets. Measurable elements include the existence of a team or department with dedicated time to meet to solve problems and support continuous learning; diversity in skills and backgrounds among team members, including representation from implementation science, quality improvement, clinical disciplines, operations, information technology, and analytics; and presence of key stakeholder relationships internal and external to the organization. Relationships occur at the patient, care team, clinic, department, and organization levels.33 Leadership is a critically important relationship to assess as they play a key role in setting expectations and communicating the value of learning.

Health information infrastructure

A key input for an LHS is an integrated, interoperable electronic health record (EHR) that supports the data requirements of multiple stakeholders, digitally captures care experiences and allows real-time access to knowledge for clinical care, research, and learning.1, 7 EHR data also enables pragmatic trial designs and other uses of “real world data” to enhance the representativeness of research trials in an LHS, and allows for longer term follow-up of treatment effects and harms at scale.34 Measurement of this construct includes the extent that the organization uses the EHR to input data, whether the EHR and other information systems have common data elements, the extent the EHR and data output interfaces with external systems like national population health registries, and whether population-level data are available at the point of care.35 Given the integral role of data and analytics to an LHS, we separated the constructs of data infrastructure (input) and data analysis (output).

Prioritization

This is the process of intentionally aligning learning activities and opportunities with strategic and operational goals across organizational levels.7 At KPWA, this is a deliberate process that occurs annually and as needed if organizational strategic priorities undergo key shifts. Alignment with strategic goals is a foundational criterion we apply when selecting new LHS projects. This construct is measured by documentation of aligned priorities.

Funding

Financial resources are essential to developing an LHS. The KPWA LHS receives programmatic funding from the health system and external funding through research-care delivery partnerships. Dedicated LHS resources are measured as total dollars committed by the organization plus total dollars committed by external sources.

Improvement infrastructure

The KPWA LHS found that an embedded QI infrastructure is a foundational LHS input. Leadership, policies and procedures to organize and facilitate improvement work across a system are essential for successfully piloting and spreading initiatives.19 The QI infrastructure must have an explicit, shared improvement methodology, approach, language, and culture; specific approach is not important. Lean, Six Sigma, and the Model for Improvement with Plan-Do-Study-Act cycles are examples of healthcare improvement approaches.36

Ethics and oversight

In an LHS, QI, clinical care and research may act synergistically, with QI and clinical care generating research, and research findings, in turn, informing clinical care and QI initiatives. Therefore, an LHS needs institutional guidance to navigate the differences, overlap, and similarities between these activities, including from Institutional Review Boards (IRBs) and compliance offices on assessing risks to participants while optimizing learning in any project on the continuum of QI, clinical care and research.7 Clear and streamlined IRB processes at KPWA make health system research nimble and efficient. This construct is measured as guidelines, procedures, and governance for LHS work.

3.1.2 Outputs

Outputs are key organization activities and deliverables that add value to care delivery. Measurement tends to be counts of deliverables. It is also important to assess the quality of these deliverables, as the quality of products is as important as quantifying production.

Environmental scanning

Internal and external assessments of the current state of an issue or practice help identify gaps and recommend best practices. Assessments can be a first step of a project or a distinct product. Environmental scans add value by identifying lessons learned, surfacing best practices, and connecting work siloed across organizational departments, other healthcare organizations, or related industries.

Evidence synthesis and translation

Health systems often struggle to interpret and apply existing evidence. Rapid literature reviews are a practical yet rigorous method to summarize evidence for a clinical topic and explain its application.20, 37 A vital part of this work is interpreting the strength of evidence so health system partners can make timely decisions about how and whether to proceed with adopting a practice.