1 INTRODUCTION

A variety of evidence-based health promotion programs are implemented in community settings in an attempt to improve population health. For instance, falls prevention programs have been introduced with the goal of reducing falls, which account for billions of dollars in health care spending in a single year.1 Chronic disease self-management programs have also shown modest reductions in emergency department (ED) visits2, 3 and hospitalization rates.2 For these and other reasons, supporters of community-based health promotion interventions have lauded the potential of such programs to benefit individuals' health, free up valuable healthcare resources, and reduce healthcare spending. However, these community-delivered programs carry their own financial costs. Research has estimated community programs to cost $350 for an individual,3 and other research has calculated that a 10% to 30% reduction in hospital admissions would be needed to make disease management programs cost effective.4 Thus, evaluating the effectiveness of community-based programs is vital to ensuring that their implementation is worth the resources invested.

Evaluating the effectiveness of evidence-based programs as routinely delivered in community settings is essential for determining their real-world impact. Randomized controlled trials are often used to assess health interventions despite possessing a number of limitations, including threats to external validity.5 By conducting a carefully planned trial in which researchers actively track participants6-8 or participants are asked to track their own health outcomes (eg, tracking falls with a fall calendar7-9 or self-reporting ER visits and hospitalization3) participants' experiences may more closely resemble that of a test participant than a community member utilizing a local program. Indeed, researchers' overemphasis on internal validity has potentially hindered our understanding of how an intervention will actually work when implemented with a new population and does not reflect the reality of likely needing to tweak interventions to better fit the context.10 Moreover, it is likely insufficient to test a community-delivered program once to determine its effectiveness. Instead, continuous monitoring of community-based programs may be necessary to continually learn about their effect and make adaptations that optimize their benefit to a changing group of participants. Indeed, because the potential value of evidence-based programs (EBPs) depends on their ability to effectively reach large numbers of people in diverse and real-world community settings, local systems may be needed to confirm that value and/or support its attainment.

2 RESEARCH INTERESTS

The present study sought to test the feasibility of using a web-based infrastructure to pragmatically assess the effectiveness of community-delivered EBPs on health outcomes. By eliminating the need for patient self-report and minimizing the research footprint, we hoped to create a learning health care system that extended beyond the typical boundaries of the healthcare system walls and from which we could iteratively learn about the effects of community-delivered EBPs and make and test changes and improvements.

3 METHODS

Two unique resources provided us with the opportunity to develop and test this infrastructure: the WellConnect network,11 which facilitates the dissemination and delivery of EBPs across community settings in Southeast Minnesota; and the Rochester Epidemiology Project (REP),12, 13 a National Institutes of Health (NIH)-funded population-based healthcare data warehouse focused on the same population. Specifically, these resources allowed us to link EBP participant records through WellConnect with health outcomes documented in REP medical records. All methods were approved by the Institutional Review Boards of Mayo Clinic and Olmsted Medical Center.

3.1 WellConnect

WellConnect (wellconnectsemn.org) was created to improve individuals' access to evidence-based health promotion programs. Healthcare providers in the southern Minnesota area were informed about WellConnect and encouraged to refer clinically appropriate patients to the programs it coordinates. All WellConnect programs were offered by community organizations at no or low cost to participants. Because programs were delivered by more than 50 different individuals across more than 10 different organizations, we had a strong rationale for exploring variation in program effects.

WellConnect offers 14 different EBPs, but for the pilot we evaluated WellConnect falls prevention and chronic disease/pain management programs because they have the largest number of participants. For falls prevention, we studied those who completed Matter of Balance,14 Stepping On,8 or Tai Ji Quan: Moving for Better Balance15 at the time of the study. These programs were conducted in a group setting and met weekly or bi-weekly. We also evaluated WellConnect's Living Well with Chronic Conditions (also known as Chronic Disease Self-Management Program16) and Living Well with Chronic Pain,17 which both met weekly.

3.2 The community data platform

To evaluate the effectiveness of WellConnect programs, we developed a secure, web-based platform that enabled WellConnect EBP leaders to enter key data elements. The development of this system is described elsewhere18, 19 and summarized in Figure 1. Briefly, the platform was able to record program types, dates and locations of delivery, leaders, and delivering organizations, as well as track participant session attendance and key demographics. Each participant's name and date of birth was used to locate their medical records from the REP. These person identifiers were chosen for matching REP medical records because they are sufficient for matching and some participants may be uncomfortable providing other identifiers (eg, social security numbers, home address).20

Diagram of learning “wellcare” system

3.3 The healthcare data warehouse

Our outcome data came from the REP, a medical record database containing comprehensive, multi-provider medical records for current and former Olmsted County, MN residents between 1966 and 2010.12 The database was subsequently expanded to include the medical records of individuals across 27 counties in southern Minnesota and western Wisconsin.13 However, there is some variance between counties in the percentage of the population whose medical records are included13 due to variation in the healthcare providers participating in the REP at different locations and their respective share of the region's healthcare data.

3.4 Linking participants records

WellConnect programs were delivered in the following 11 counties: Wabasha, Goodhue, Winona, Houston, Olmsted, Steel, Rice, Dodge, Mower, Freeborn, and Fillmore. We encouraged EBP program leaders to obtain written consent to share attendance statuses for the purpose of research from all program participants as part of their normal record-keeping process (eg, via a one-page consent form included in a folder of program materials). Consent status (yes vs no) could then be documented in the platform. Paper-based records of all information (eg, participant names, session attendance, consent status) were also kept via routine processes by the Area Agency on Aging. We obtained these paper records and used them to validate and update the online database before exporting all data for analyses. We filtered the data to identify participants who consented and searched the REP for person IDs that matched the identity of these individuals.

3.5 Assessing participant outcomes

For participants for whom we could confidently identify a REP record, we identified two matched—by age, sex, current county of residence, and number of ED visits and hospitalizations in the year prior—controls. For falls program participants, we additionally, matched controls on whether they had a documented fall (based on ICD 9/10 codes) in the year prior. Methods for determining fall rates in this way are described elsewhere,21 but are known to grossly underestimate. We then used these controls to explore the feasibility of using the infrastructure to evaluate program effectiveness in a small pilot. Specifically, we compared the incidence of hospital admissions, by pulling encounter data using clinic numbers, for all EBP participants and falls for fall prevention program participants at 1 year of follow-up and against controls. All outcomes were binary. We calculated a statistical power of 83%, contingent upon us having 542 EBP participants, 2 matched controls per participant, and at least 25% of controls being admitted to the ER at 6 months.

Descriptive characteristics were reported with frequency percentages for categorical variables and mean and standard deviations for continuous variables. We used two separate logistic regressions to explore differences between falls prevention EBP participants and matched controls on the likelihood of (a) having a fall and (b) being admitted to the hospital or ED over 1 year of follow-up. An additional logistic regression explored differences in the likelihood of being admitted to the hospital or ED between chronic disease or pain management EBP participants and their matched controls over 1 year of follow-up. All analyses adjusted for gender and age. A P-value of .05 was used as the threshold for statistical significance for all models. Data was analyzed using SAS statistical software (SAS version 9.4; SAS Institute Inc).

4 RESULTS 4.1 Feasibility

Of the 737 EBP workshop signups where consent was provided to share attendance information for research, we were able to link medical records from the REP for 572 (77.6%) cases. A number of community leaders did not enter EBP participant information into the web-based platform, but we obtained this information from paper-based records kept by the Area Agency on Ageing. We entered these records into our platform ourselves, which created an extra step but did not result in data loss. Thus, we were able to link medical records for the majority of participants who consented. However, obtaining consent from participants to share their medical records proved more challenging; we received consent for only a little over half (56.5%) of the 1304 WellConnect EBP signups at the time of the pilot. This presents a threat to the feasibility of our platform because obtaining outcome data for only a subset of study participants could bias the findings on program effectiveness. However, EBP leaders were not researchers with experience consenting participants and did not receive standardized instructions on how to elicit participant consent. As a result, they may have only briefly mentioned the consent form in the packet, de-emphasizing its importance, or struggled explaining how participants' medical records would be used, perhaps contributing to the lower consent rate.

4.2 Participant outcomes

About 56% (n = 322) of program signups for which we had obtained consent and we were able to link medical records, were program non-completers (missed two or more sessions). These cases were excluded from the analyses to ensure programs were evaluated on their own merit in a best-case scenario and to align with published understanding of necessary dose. An additional n = 20 cases were found to be duplicates and dropped, n = 25 participants were dropped because they had participated in multiple EBPs simultaneously, and n = 9 diabetes EBP participants were dropped because there were not enough diabetes EBP participants for analyses. This left n = 99 falls prevention and n = 97 chronic disease/pain management EBP completers. See Table 1 for descriptive characteristics of the falls prevention and chronic disease/pain management EBP participants and their matched controls.

TABLE 1. Descriptive characteristics Falls prevention EBP Chronic disease pain management EBP Case (n = 99) Control (n = 198) Case (n = 97) Control (n = 194) Gender Male 17 (17.2%) 34(17.2%) 18 (18.6%) 36 (18.6%) Female 82 (82.8%) 164(82.8%) 79 (81.4%) 158 (81.4%) Race White 93 (93.9%) 192(97.0%) 87 (89.7%) 189 (97.4%) Other 1 (1.0%) 3(1.5%) 3 (3.1%) 2 (1.0%) Unknown 5 (5.1%) 3(1.5%) 7 (7.2%) 3 (1.5%) Age Mean ± SD 76.4 ± 8.3 77.2 ± 8.5 72.2 ± 12.5 72.6 ± 12.3 Min, Max 54, 101 54, 103 27,94 28, 94

Findings revealed that at 1 year follow-up, four falls prevention EBP participants (4% of participants) and three matched controls (1.5% of controls) had at least one fall documented in their medical record, and 19 of the falls prevention EBP participants (19.2% of participants) and 52 of the matched controls (26.3% of controls) had at least one documented hospital admission. Of those who participated in chronic disease or pain management EBP, 16 EBP participants (16.5% of participants) and 38 matched controls (19.6% of controls) had at least one documented hospital admission at 1 year follow-up. None of these differences reached statistical significance when adjusting for covariates (see Table 2).

TABLE 2. Differences in Health Outcomes between EBP Participants and Matched Controls Falls prevention EBP vs controls (N = 297) Chronic disease/pain management EBP vs controls (N = 291) Fall OR(95%CI) ED/hospital admittance OR(95%CI) ED/hospital admittance OR(95%CI) Group status Control (ref. group) - - - EBP participants 1.21(0.18, 7.96) 0.66(0.36, 1.19) 0.81(0.43, 1.54) Gender Male (ref. group) - - - Female 3.41(0.70, 16.60) 0.82(0.40, 1.68) 0.83(0.39, 1.77) Age 1.60(1.04, 1.28) 0.98(0.95, 1.02) 0.99(0.97, 1.02) Note: *P < .05, **P < .01. 5 DISCUSSION 5.1 Our findings

When a community-based organization implements an EBP that previously worked in one community,8 it may not work in a new community, leading to retrospective attempts to identify reasons for the reduced effectiveness.22 This method requires heavy involvement from research personnel and may disrupt the natural operation of community organizations delivering these programs. The present study created and piloted a novel web-based infrastructure that was able to unobtrusively and pragmatically gather information on community-based EBPs that aimed to improve health outcomes, which can then be used to refine program delivery. This was accomplished without embedding research personnel within community organizations, and without researchers having to recruit or collect information from community members who utilized these community programs. Even without using active recruitment methods, for example, we were able to confirm that the demographics of those who completed a WellConnect falls prevention program were similar to those in whom the program was known to work (eg, 74% female and mean age of 78 years old).8 We did not, however, have a mechanism for unobtrusively collecting process measures or evaluating the fidelity of program delivery, due to the difficulty of passively collecting this data. This may be necessary to fully realize a learning “wellcare” system in which community organizations continually learn from and modify their programs. However, researchers would likely need closer involvement with community programs to evaluate these implementation outcomes. In that regard, we feel our pilot study established a proof-of-concept and has illustrated clear areas for improvement.

There were no statistically significant differences in health outcomes between WellConnect program participants and matched controls. Nonetheless, falls prevention EBP participants demonstrated a 34% decreased likelihood of being admitted to the ED or hospital at 1 year of follow-up and chronic disease/pain management EBP participants demonstrated a 19% decreased likelihood compared to matched controls. This is similar to published reductions in the likelihood of ED (32%) and hospital (28%) admittance found for chronic disease management program participants.23

5.2 Limitations

Despite the strengths of the web-based infrastructure, there are several limitations to its generalizability and areas for improvement remain. First, our novel infrastructure relies on linkages to the REP (or a similar medical record database). While Electronic Health Records (EHRs) have proliferated making data linkage more feasible, population-based data warehouses that pool the records of multiple providers remain uncommon. Even the REP database is not fully comprehensive; some individuals choose to “opt” out of sharing their medical record information,12 limiting the generalizability of REP findings to other populations. Further, because not all healthcare providers participate in the REP,12 if an individual included in the REP seeks medical care from a non-participating provider, that medical event will not be documented in their REP medical record. Thus, it is possible that our infrastructure (and locating outcome data in medical records in general) may provide underestimates of health outcomes if some medical records are not properly identified or linked. Related to this, the lack of statistically significant findings on fall events may have been due to the small sample size, the low number of fall events documented in the medical record, and/or from the comparatively low overall incidence of falls in our pilot sample.8 It is possible that those who were well enough to complete a community-based falls prevention program were not at particularly high-risk for falls, explaining the lower fall rates observed. Furthermore, the incidence of falls in our pilot study may have been an underestimate if some participants experienced a fall but did not seek medical attention. Research suggests that only 31.3% of falls reported by those 65 or older either impaired function for a day or longer or prompted seeking medical care,24 indicating the majority of falls never become part of the medical record. However, because EBP participants and matched controls are both impacted by these potential limitations equally, any misclassification is likely to be non-differential, which would bias towards to the null. Nonetheless, we may be missing valuable data on program effectiveness by not having participants self-report on falls or other health outcomes not documented in the medical record. However, collecting this data would interfere with the passive nature of this research, decreasing the ecological validity of community program participation, which poses a dilemma that this type of research will continue to grapple with.

5.3 Future directions

Despite these weaknesses, our platform is a promising innovation for assessing the outcomes of community implementation of EBPs and for conducting natural experiments. Future developments of the platform may involve strategies to improve completeness of data capture by increasing information sharing consent rates and program completion. We are considering highlighting the potential of sharing data to benefitting those with similar medical concerns as a way of encouraging consent.20 Additionally, we are revising our consent strategy to prioritize an opt out approach. Further, because some data was lost to program non-completion, we will need to better explore reasons for non-completion and identify potential ways to target these barriers. Data capture may also be improved by encouraging more providers to contribute medical records to the REP (we were unable to locate medical records forabout 22% of cases in which we had received consent). Efforts to enhance our infrastructure's ability to track program fidelity and expand the scope of information it collects in regards to community contributions to health beyond EBPs are also warranted. Clinical and research endeavors that seek to replicate our system or similar approaches in other locations should also be pursued to assess the viability of its intentions more generally.

6 CONCLUSION

This pilot study demonstrated the feasibility of using a novel web-based platform to assess the relationship between participation in community-based programs and the receipt of formal healthcare services. We were successful in both organizing a group of community organizations, known as WellConnect, to utilize our infrastructure and linking health outcome data from medical records. Moreover, our pilot study demonstrates the ability to bridge knowledge gained from community programs and the healthcare system to promote community wellness, but suggests that obtaining consent from participants to link their medical records presents a challenge. Additionally, using our infrastructure requires a large upfront resource investment (e.g., finding access to a large medical record database and identifying a statistician to link attendance information to medical records). However, once these resources are in place, this will likely create a sustainable infrastructure to monitor program effectiveness and the effectiveness of adaptations shaped by this feedback.

ACKNOWLEDGEMENTS

This project was supported by the Mayo Clinic Division of Community Internal Medicine and by CTSA Grant Number UL1 TR002377 from the National Center for Advancing Translational Sciences (NCATS). Its contents are solely the responsibility of the authors and do not necessarily represent the official views of the NIH.

CONFLICT OF INTEREST

The authors have no relevant conflicts of interest to disclose.