Study Design

This study comprises one segment of a project assessing predictors and consequences of VHA BHP burnout.18, 20 We used facility-level quality metrics as outcomes and burnout as primary predictors. A “station” (STA3N) within VHA represents a parent facility and may have several subsidiary medical centers or community-based outpatient clinics assigned to it. Since participants provided anonymous responses, we cannot link data by respondent within any data sources or between surveys. VA Ann Arbor Healthcare System Institutional Review Board approved this study.

Data Sources

We used 2014–2019 data from Annual All Employee Survey (AES) and Mental Health Provider Survey (MHPS); 2015–2019 facility-level Mental Health Onboard Clinical (MHOC) staffing and productivity data; and 2015–2019 MH-SAIL data. After merging sources, the study included 127 out of 138 (92%) VHA parent facilities with available data.

AES

National Center for Organizational Development (NCOD) administers AES to all VHA employees annually to assess workplace perceptions and satisfaction. Further information on AES creation, measures, and how it informs VHA appear elsewhere.21 Since 2001, AES represent best practices among large organization surveys.22 All AES responses remain anonymous. We included BHPs: psychologists, psychiatrists, social workers. During the study period, AES had response rates of 54% in psychiatrists, 66% in psychologists, and 67% in social workers.20

MHPS

Office of Mental Health and Suicide Prevention (OMHSP) invites all VHA BHPs to complete the MHPS annually to assess perceptions about access to and quality of mental health care, and job satisfaction.23 Analyses found MHPS data reliable, valid, and consistent.24 The MHPS response rate during the study period exceeded 50%.20

MHOC

OMHSP developed a staffing model that estimates full-time equivalent (FTE) mental health staff per 1000 Veterans treated in outpatient mental health settings, a population-based measure (staffing ratio).25 MHOC includes a measure of provider productivity calculated as the sum of work Relative Value Units (wRVUs) divided by time spent providing direct clinical care in outpatient mental health settings (productivity).26

CDW

We used VA Corporate Data Warehouse (CDW) to create a facility indicator of rural/urban location and a three-part facility complexity measure. Complexity levels include high (high-volume, high-risk patients, most complex clinical programs, large research and teaching programs), medium (medium volume, low-risk patients, few complex clinical programs, small or no research and teaching programs), and low (low volume, low-risk patients, with few or no complex clinical programs, small or no research and teaching programs).

MH-SAIL

In 2010, VHA implemented the SAIL monitoring system to provide VHA management with high-level indicators of health care quality.27 MH-SAIL incorporates a composite of three component measures, each of which represents a composite of constituent measures (see Appendix Table 1). Three components include population coverage, continuity of care, and experience of care. Experience of care includes four provider (collaborative MH care; job satisfaction; quality of MH care; timely access to MH care) and two patient experience subcomponents (MH appointment access; patient-centered MH care). VHA developed components tailored to its intended coverage, available data, and candidate measures identified during selection and refinement. Each component represents measures with moderate to high internal consistency.24

Study MeasuresDependent Variables

We used four MH-SAIL metrics as outcomes.

Population Coverage

An objective measure representing access to care, which combines 16 individual metrics (constituent items) with denominators representing the number of Veterans experiencing specific diagnoses and numerators representing receiving targeted services, treatments, and/or visits.

Continuity of Care

An objective measure combining 11 individual metrics with denominators representing the number of Veterans experiencing specific diagnoses and treatments for and numerators representing continuity of care such as number of follow-up visits within a specified period or amount of continuous medication coverage.

Experience of Care

A subjective measure that includes both provider and patient perspectives combining 32 individual survey responses including provider responses assessing collaborative MH care (6 items), job satisfaction (2 items), quality of MH care (5 items), timely access to MH care (6 items), and Veteran responses assessing MH appointment access (5 items) and patient-centered MH care (8 items).

We used the three domain scores (population average, continuity of care, experience of care) generated by VHA developers of the tracking system every quarter. Within experience of care domain, we also used subdomain scores of provider satisfaction and patient satisfaction. The domain scores each use weighted averages of standardized constituent items where each item’s scores represent quarterly changes from the score of the last quarter of the prior fiscal year within each facility, divided by the standard deviation of the prior year last quarter facility score, and thus have a mean of 0 and standard deviation of 1.24 The standardization combines constituent items with different denominators and statistical distributions into like units to generate each domain scores. The domain scores indicate overall direction of change in a facility’s performance for the specific domain within-facilities.24

Overall Mental Health

This measure represents an overall measure calculated as the equally weighted averages of the two objective and one subjective domain scores.24

Key Independent Variable: Provider Burnout

For AES and MHPS, we defined employee burnout as a dichotomous variable using validated approaches to define burnout.20, 28 We obtained the facility burnout percentage for each survey by averaging dichotomous burnout data among facility survey responses. We analyzed burnout data from AES and MHPS separately because we cannot identify respondents in each survey or link participants who completed the two sets of measures.

AES

We classified whether respondents reported burnout according to methods used by other VHA researchers.28 The approach used two burnout questions: emotional exhaustion (“I feel burned out from my work”) and depersonalization (“I worry that this job is hardening me emotionally”). Each of these two burnout questions had a 7-point response scale (1 = never; 2 = a few times a year or less; 3 = once a month or less; 4 = a few times a month; 5 = once a week; 6 = a few times a week; 7 = every day). We generated a dichotomous variable such that if the respondent answered either question with 5 or higher (once a week or higher frequency), we classified the response as endorsing burnout; otherwise, we classified respondents as not endorsing burnout, as in our prior study.28

MHPS

We generated a dichotomous variable to classify respondent burnout using the sole burnout question of “Overall, based on your definition of burnout, how would you rate your level of burnout?” The response options from 1 to 5 appeared as follows: 1 = I enjoy my work. I have no symptoms of burnout; 2 = Occasionally I am under stress, and I don't always have as much energy as I once did, but I don't feel burned out; 3 = I am definitely burning out and have one or more symptoms of burnout, such as physical and emotional exhaustion; 4 = The symptoms of burnout that I’m experiencing won’t go away. I think about frustration at work a lot; 5 = I feel completely burned out and often wonder if I can go on. I am at the point where I may need some changes or may need to seek some sort of help. We generated the dichotomous burnout variable by response of ≥ 3. Our prior work indicated that facility-level MHPS burnout rate using ≥ 3 as the cutoff showed the highest correlation to facility-level burnout rate in AES across yearly data from 2015 to 2018.20

MHOC

We used two facility-level variables (staffing ratio and productivity) as possible predictors of the relationship between self-reported work-environment characteristics and burnout. Details outlining the purpose, origins, and definitions of these metrics appear elsewhere.25, 26, 29

Statistical Analysis

We summarized facility-level characteristics (annually), burnout (annually), and MH-SAIL domain scores (annually by averaging four quarterly scores). As annual burnout percentages represent summary data for BHPs who responded to the burnout items and do not include non-responders to the burnout items, we summarized the yearly burnout percentages as (1) crude average of the facility burnout percentages among BHPs, (2) average weighted by the number of facility survey responders, and (3) average weighted by inverse of the facility response rate of burnout items, using this final approach in our adjusted models.

We assessed relationships between burnout and MH-SAIL outcomes using multiple regression analysis with facility-level prior year burnout percentages among BHPs as predictors and weighted by the number of facility survey responders. We examined impact of prior year burnout on subsequent year MH-SAIL outcomes to disentangle temporal ordering of provider burnout and outcomes. We repeated analyses using burnout percentages based on yearly AES and on MHPS separately to assess consistency. For each year, we estimated raw and covariate adjusted facility-level burnout effect on each of the four MH-SAIL composite outcomes. For meaningful interpretations of the burnout regression coefficient, we divided facility burnout percentage by 5. A one-unit increment in burnout corresponded to a 5% increment in burnout. In adjusted models, we included as covariates facility complexity, rurality, staffing ratio, and productivity. Finally, we pooled data across years and obtained a summary burnout effect on each MH-SAIL outcome (the four composite measures and six composite experience of care measures) using generalized linear models with generalized estimating equations (GEE) to account for repeated data over years within each facility, adjusting for covariates and year.