IntroductionBurnout and the Medical Profession

The medical profession is facing a burnout crisis []. Prevalence estimates among physicians range between 37% [] and 80% [], and the enduring challenges associated with the COVID-19 pandemic have exacerbated this existing problem [,]. Burnout is a psychological outcome of exposure to work-related stress characterized by varying degrees of exhaustion (feelings of overload and depleted emotional energy), cynicism (detachment, withdrawal behaviors, and diminished idealism), and inefficacy (the perception that one’s own work performance lacks quality and value) [,]. Although previously believed to emerge during the later stages of a medical career [], a high global prevalence among medical students (44.2% []) is indicative of earlier onset. Increasingly, burnout is arising in response to pervasive imbalances between medical education stressors and a student’s coping resources, and this can persist throughout an individual’s professional life [,]. Individuals affected by burnout are at risk of a range of associated outcomes that extend beyond the work context, including poor physical [,] and psychological health (including diminished well-being [], depression [,], anxiety [], and suicide [,]). Furthermore, burnout impedes medical service delivery and productivity through its adverse impact on work engagement (eg, reduced participation) [] and performance (eg, increased medical errors and diminished quality of patient care [,]).

Developing effective strategies to address the problem of physician burnout and its associated outcomes was an identified research priority before the pandemic [,], with a particular emphasis on early intervention strategies that could facilitate prevention [,-]. However, progress toward this agenda was disrupted by the emergence of COVID-19, which required the medical profession to rapidly prioritize and respond to the ensuing public health crisis, often under conditions of uncertainty and insufficient resources []. The persistent stress of the pandemic has contributed to an increase in rates of physician burnout [] and declining mental health among medical students, to the extent that many report reconsidering their decision to pursue a medical career []. There are growing concerns that the potential endurance of these adverse mental health impacts beyond the pandemic could further diminish the resilience of the medical workforce and health care systems, adding to the urgency of identifying and deploying effective interventions [].

Adaptive Psychological Skill Training for Medical Students

Organizational interventions that modify external stressors within the work or study environment (eg, inadequate resources, excessive workloads, and time pressures) have demonstrated benefits [-] and are essential to burnout prevention strategies []. However, the pandemic has highlighted that the modifiability of some of these factors may be limited during extended periods of crisis when health care resources are stretched beyond capacity. Medical students in a recent study reported that pandemic-related stressors contributed to elevated burnout and diminished mental health, some of which were less modifiable under the circumstances (eg, web-based learning fatigue, restricted opportunities for clinical experience, and mandatory isolation) than others (eg, quality of web-based learning) [,]. Furthermore, physicians and medical students encounter unmodifiable demands inherent to their training and work, resulting in unavoidable contact with certain risk factors during the normal course of their careers [] (eg, academic pressures, exposure to death and dying [,], and role responsibility []).

Psychological and behavioral responses to demands and stressors play a role in burnout development among medical students, whose risk of burnout is almost doubled when maladaptive coping patterns are adopted [-]. There is recent evidence suggesting that, in addition to its augmentation of external stressors, the pandemic has adversely affected medical students’ psychological resources for coping with these challenges []. Maladaptive coping patterns are unlikely to spontaneously improve over time without intervention [], which may contribute to the increasing risk of burnout that medical students face as they progress through training and into their careers [,,]. Individual-level interventions have the potential to buffer against burnout and other psychological ill health outcomes by training modifiable cognitive, emotional, and behavioral skills that can facilitate adaptive responses to unmodifiable contextual stressors [,,,]. There is growing support for the implementation of such interventions during undergraduate medical training, which is recognized as a critical stage of learning and career preparation [,,]. Physicians experiencing burnout have proposed that learning self-care should be prioritized equally with clinical skills during medical education []. By assisting students in developing adaptive coping repertoires, individual-level interventions can pre-emptively prepare them to respond to the inevitable stressors of medical education and future practice in ways that support their psychological health and well-being [,,]. This early intervention strategy could offer longer-term benefits for burnout prevention within the medical profession [,,,,].

Intervention Model: Psychological Flexibility

Despite the considerable increase in the number of studies investigating such interventions in recent years, systematic reviews have highlighted the need for more rigorous studies identifying which adaptive psychological skill sets can be trained to produce optimal improvements in burnout and well-being outcomes among medical students [,]. Psychological flexibility is a set of adaptive cognitive and behavioral skills that is a promising intervention target [,-]. The psychological flexibility model encompasses 6 modifiable flexibility (and corresponding inflexibility) processes: present-moment awareness (nonawareness of present moment), experiential acceptance (experiential avoidance), cognitive defusion (cognitive fusion), self-as-context (self-as-content), contact with values (lack of contact with values), and committed action toward values (inaction) [,]. When faced with challenging situations or uncomfortable internal experiences, individuals who are high in psychological flexibility tend to respond in ways that are effective in the moment and supportive of their broader well-being [,], including bringing conscious awareness and openness to the conditions of their present-moment experiences and purposefully committing to effective actions that align with personally held values []. Conversely, psychological inflexibility manifests as behavioral rigidity in response to internal (eg, thoughts, emotions, and physical sensations) experiences, which can have an adverse impact on an individual’s capacity to adapt and function in psychologically healthy ways []. An individual can learn to improve their psychological flexibility skills using a range of intervention approaches, including Acceptance and Commitment Training (ACT) [,]. Rather than focusing on directly modifying psychological symptoms, ACT aims to increase psychological flexibility and decrease psychological inflexibility processes in the service of expanding adaptive, values-based behavioral repertoires []. This can alter the way in which an individual responds to stressful experiences, producing secondary benefits across a wide range of psychological and organizational variables [-].

Within the broader literature, psychological flexibility is associated with well-being [,] and has been shown to protect against burnout and adverse mental health outcomes in stressful situations [,,] (including depression and anxiety []). Interventions that train individuals to develop and strengthen their psychological flexibility skills are effective in reducing work-related stress and burnout severity [,,-] and improving well-being [,,]. Mediation studies demonstrate that ACT interventions exert their beneficial impacts on burnout and well-being by improving an individual’s psychological flexibility [], suggesting that this skill set is a mechanism of change for these outcomes. In a longitudinal ACT intervention study, improvements in psychological flexibility mediated reductions in exhaustion [], and this prevented the later development of cynicism []. This is an important finding with respect to the potential burnout prevention benefits of psychological flexibility training among medical students, for whom exhaustion is the most prevalent factor and is considered a foundational manifestation of burnout [,].

There is early evidence suggesting that psychological flexibility skills may function as important personal resources in the medical profession []. Low psychological flexibility is associated with burnout risk among medical students, physicians [], and resident physicians []. Medical students who report low psychological flexibility also demonstrate diminished satisfaction with life and greater personal distress when seeing others in harm, which may increase burnout risk during their careers []. Furthermore, burnout risk is higher among medical students who engage in experiential avoidance [,] or non–values-based actions []. Conversely, higher psychological flexibility was recently found to predict lower burnout among medical students and physicians during the pandemic []. There is minimal research examining the benefits of psychological flexibility skill training interventions for medical students. A recent study found that distressed medical students who completed ACT training as a requirement of their undergraduate medical curriculum demonstrated improved burnout outcomes, but this study did not include a control group []. A small semiexperimental study demonstrated improvements in well-being and psychological distress among female medical students in Iran following completion of an ACT intervention []. Further rigorous efficacy studies are needed to evaluate whether training medical students in psychological flexibility skills improves burnout and psychological well-being outcomes.

App-Based Intervention DeliveryAccessibility

A recent systematic review of individual resource-building interventions for medical students highlighted the need for research evaluating the effectiveness of nontraditional delivery methods such as smartphone apps []. As medical students use smartphones frequently [], stand-alone app–delivered interventions have the potential for cost-effective scalability [,] and can offset known accessibility barriers by providing anonymous and private access to medical students concerned about mental health stigma [,,]. Stand-alone apps offer accessibility in times when face-to-face delivery is not an option, such as during pandemic-related lockdowns [], and can deliver brief training components at convenient times for those with busy schedules [,]. Psychological skill generalization may also be enhanced because of the accessibility of training opportunities in everyday life situations []. However, feasibility trials of app-based psychological interventions for medical students also indicate that maintaining engagement in this accessible medium can be challenging [,].

Individualization

App technologies have been identified as important to advance key research priorities within the psychological flexibility literature because of their potential to facilitate methodologies that accommodate individual heterogeneity and enhance intervention precision []. Although the benefits of ACT have been assessed in >1000 randomized controlled trials (RCTs) [], traditional approaches evaluating the effectiveness of generic interventions by comparing aggregated group data are limiting as they overlook important information regarding individual differences in training needs and outcomes [-]. Generalized deployment of interventions found to be effective at the group level is likely to result in some individuals not receiving the type or amount of training they need and others receiving more than necessary [,]. With respect to psychological flexibility processes, research suggests that individuals might require training in different skills at different times and in different situations []. Although high psychological flexibility is often associated with low psychological inflexibility (and vice versa), several distinct and more complex profiles have recently been observed [], indicating that flexibility and inflexibility processes do not necessarily exist at opposite ends of a spectrum but may vary temporally and contextually within individuals [,,]. Thus, there is a growing focus on individualized approaches in which intervention decisions are driven by an individual participant’s identified needs and skill deficits in varying moments and contexts [,]. Apps have the potential to facilitate both the real-time ecological momentary assessment (EMA) [] of these needs in a participant’s everyday life and the targeted delivery of just-in-time adaptive intervention components that align with these needs []. Importantly, as an individualized approach adjusts for heterogeneity at the intervention level, an RCT study design can be used to assess its treatment utility over a nonindividualized approach [].

As ACT is a theory-driven intervention with practical training components that have clear functional links to the corresponding core process within the psychological flexibility model [,,], it is well suited to individualized delivery via an app []. Levin et al [] demonstrated a simple EMA strategy for identifying a participant’s present-moment psychological flexibility training needs, involving a brief subjective assessment of which process an individual felt they were experiencing the most difficulty with each time they accessed an app-delivered ACT intervention. This information was used to individualize the training by delivering a practical ACT skill activity that aligned with the identified psychological flexibility process at that moment. Using a 3-arm RCT, the researchers demonstrated that, compared with a nonindividualized ACT app and a nontreatment group, university students who engaged with the individualized app demonstrated statistically significantly greater improvements in psychological distress and well-being. Empirical evaluation of individualized ACT interventions is in its early stages, and more research is needed to examine effectiveness and advance the development of individualization methodologies [,,]. To date, no studies have examined the potential burnout and well-being benefits of training medical students in psychological flexibility skills using individualized or nonindividualized ACT apps.

Study Aims

The aim of this RCT was to evaluate the effectiveness of an app-delivered ACT intervention for medical students with respect to burnout (exhaustion [primary outcome], cynicism, and academic efficacy), well-being, and psychological flexibility and inflexibility outcomes. We hypothesized that medical students who engaged with either an individualized or nonindividualized version of the ACT app would demonstrate greater postintervention improvements in outcomes than those in a waitlist group and that intervention effects would be greater for the individualized group than for the nonindividualized group. We aimed to examine whether any observed postintervention improvements in burnout or well-being would be mediated by improvements in psychological flexibility and inflexibility. Furthermore, we aimed to explore whether engaging in either version of the ACT app would improve other relevant secondary psychological outcomes (stress, depression, and anxiety).

MethodsEthics Approval

Ethics approval for this study was granted by the University of Newcastle Human Research Ethics Committee on January 21, 2021 (approval ID: H-2020-0311), and ratified by the University of New England Human Research Ethics Committee on February 11, 2021.

Trial Design

This study was a 3-arm, parallel RCT of a 2-stage psychological flexibility skill training app for medical student burnout and well-being. Randomization was performed within the app using a 1:1:1 allocation ratio and a simple randomization procedure where each student had a 1 in 3 chance of allocation to each intervention arm (individualized intervention, nonindividualized intervention, and waiting list). The inclusion of the nonindividualized group was an important element of this research design as it provided a generic ACT intervention control condition against which to evaluate the potential relative benefits (or “treatment utility” []) of the individualized approach adopted []. The functional limitations of the app meant that it was not possible to stratify randomization by participant baseline characteristics. Allocation was blinded for participants assigned to the individualized and nonindividualized intervention groups. Blinding was not possible for the waitlist group.

Recruitment and Study Setting

Recruitment was conducted for 6 weeks during August 2021 and September 2021 (refer to for the participant flow diagram). This unintentionally coincided with a mandated lockdown because of a regional COVID-19 outbreak. The sampling frame was students enrolled in first, second, fourth, and fifth years of the Joint Medical Program (JMP) at the University of Newcastle or the University of New England, Australia (N=778). During the first and second years of the JMP, students predominantly engage in in-class academic learning, whereas the fourth and fifth years involve a stronger focus on applied clinical training. Students enrolled in the third year of the JMP were not invited to participate in this study as they had previously taken part in a feasibility trial of the app.

‎Figure 1. Participant flow diagram. II: individualized intervention; NI: nonindividualized intervention; WL: waiting list.

Students were invited to participate in a trial to assess the efficacy of a psychological flexibility skill training app for well-being and burnout prevention. An invitation email was sent to students’ university accounts from a JMP administrative account approximately every 2 weeks during the recruitment period, and a member of the research team also delivered a verbal invitation at the end of web-based classes. Students were provided with a URL and QR code that gave them access to the enrollment website (located on a secure web-based survey and database platform, REDCap [Research Electronic Data Capture; Vanderbilt University] [,], hosted at Hunter Medical Research Institute) during the recruitment period.

Students who accessed the enrollment website were assessed for eligibility and were eligible for inclusion if they had regular access to a reliable internet connection and an electronic device compatible with app use (smartphone or tablet). There were no ineligibility criteria. Participation was voluntary, confidential, and independent of students’ academic programs. Students were required to read a participant information statement before deciding whether to enroll in the study. This included information about the purpose of the study, time required to complete outcome measures (approximately 10 minutes at each time point), intervention and study flow, privacy and data storage, and withdrawal processes. Students who elected to participate registered their consent through an e-consent process. Following completion of a brief self-report demographic assessment (baseline participant characteristics), enrolled students were provided with links to download the app via the Play Store (Android) and App Store (Apple). The app was password-protected to ensure that only study participants had access to the outcome measures (closed surveys) and intervention.

Data Privacy

To ensure the privacy and anonymity of participant data, 2 password-protected databases were used to store personal and demographic information (in an identifiable “Participant Information Database”) separately from the outcome data (stored in an anonymous “Study Database”). Outcome data were deidentified, with participant ID used as a linkage key. Only the lead author had access to the participant information database. Members of the research team did not have access to identifying participant information. To protect privacy while using the app, students input their unique participant ID as their username when registering and accessing their account.

ProceduresOverview

The study period for each participant commenced when they downloaded the app and registered their account. All further assessment, intervention, and data collection procedures were conducted within the app, which students accessed independently during the study period. The intervention and assessment components were resumable, allowing students to engage with the app at their own pace. Participants first completed baseline psychological outcome measures followed by a brief (10 minutes) onboarding session, which outlined how to use the app and introduced psychological flexibility as an adaptive behavioral skill set that could help students do what is important to them while effectively navigating life’s challenges. Students were then randomized into 1 of the 3 intervention arms, after which the app was programmed to deliver the intervention pathway associated with their group allocation. The individualized and nonindividualized groups were given access to the psychological flexibility training intervention via the app for 5 weeks. For students assigned to the waiting list, access to app content was disabled for 5 weeks. After the study period, all 3 groups were asked to complete the postintervention outcome measures. Upon completion, waitlist participants were given access to the individualized version of the app, and intervention group participants were given continued access to their allocated version. All students who completed the postintervention outcome measures were given an Aus $30 (US $20.70) digital gift voucher to acknowledge the time commitment involved in participation.

The 5-week intervention duration was set in consultation with the participating medical schools. This provided students with the opportunity to participate in the study during the semester while reducing participation burden by ensuring that the timing of outcome measures did not correspond with critical assessment periods. This duration was considered appropriate as Levin et al [] observed intervention effects for a similar 4-week program, and feasibility trialing of the current app demonstrated that medical students engaged most frequently during the early weeks of access [].

InterventionOverview

The intervention was an ACT-based psychological flexibility [] training program delivered via a smartphone app (“BiSi: Build it. Sustain it.”) created for this study. A more detailed protocol for this study is available elsewhere []. BiSi was developed by clinical psychologists (ED and BK) with extensive experience in the psychological flexibility model and involved the adaptation of existing ACT concepts and training activities to suit the app-based context and target participant group. The intervention was delivered in 2 stages.

Stage 1: Learn the Concepts (Introductory Module)

Stage 1 involved the delivery of an introductory module (“Learn the Concepts”), which was identical for the individualized and nonindividualized groups. The purpose of the module was to familiarize students with the psychological flexibility model and its potential benefits and provide a conceptual framework for understanding the stage 2 experiential skill training activities. Students could complete the module over multiple sittings (total completion time of approximately 60 minutes depending on individual pace). The module comprised 7 sections (<10 min each) presented in a fixed order. Students were required to complete each section before progressing to the next. Section 1 provided psychoeducation about burnout (focusing on destigmatizing burnout-related experiences and how to recognize the signs) and well-being (including the importance of choosing actions that involve the consideration of personal well-being). Sections 2 to 7 provided education about each psychological flexibility and inflexibility process and outlined how psychological flexibility skills could be implemented to support well-being and protect against burnout. Although the module was primarily conceptual, experiential components encouraged self-reflection and provided opportunities to practice each skill set (eg, personal values identification, mindfulness, experiential acceptance, thought defusion, perspective shifting, and values-based goal setting). Written and audio versions of the psychoeducational content were provided, along with accompanying images. Some skill activities were presented in either written or audio format depending on which translated best to the app-based delivery mode. Completion of the stage 1 introductory module unlocked students’ access to the stage 2 on-demand skill-training dashboard.

Stage 2: Learn the Behaviors (On-Demand Skill Training)

Stage 2 provided access to a library of short (3 to 8 minutes) experiential psychological flexibility skill activities that students could practice at any time (on demand). Each of the 6 psychological flexibility processes had its own dashboard where all ACT training activities for that skill set were displayed on a list (20 activities per process; 120 in total). Although most activities were presented in audio format with accompanying images, some included written components and items that required the user to input a response.

During stage 2, students from both intervention groups were presented with a main dashboard (home screen) each time they opened the app. To access the psychological flexibility skill activities, students selected “Practice a Skill” on the home screen and were presented with a single-item EMA screening question (“Which of the following are you having the most difficulty with today?”) to identify which psychological flexibility skill set might be most relevant to their training needs on that occasion (adapted from Levin et al []). Students selected 1 of 6 response options, each of which corresponded to one of the 6 psychological flexibility processes: “Struggling with your feelings” (acceptance), “Unable to do what matters to you” (committed action), “Stuck in your thoughts” (defusion), “Stuck in autopilot or struggling to stay in the present moment” (present-moment awareness), “Disconnected from a sense of meaning or purpose” (values), and “Stuck in stories about who you are or who you should be” (self-as-context).

The EMA screening question was used to tailor the intervention to the training needs of students in the individualized group. Upon completion of the screening question, participants in the individualized group were presented with the dashboard corresponding to the psychological flexibility skill set they had identified as having the most difficulty with on that occasion (eg, if a student reported feeling disconnected from a sense of meaning or purpose, the app displayed the values dashboard). This allowed students in the individualized group to practice a skill targeted to the area of identified need each time they accessed the app. For the nonindividualized group, students’ responses to the screening question had no impact on the skill set in which they received training. After responding to the screening question, students in the nonindividualized group were presented with 1 of the 6 psychological flexibility dashboards at random (simple randomization; each process had an equal chance of selection). Students were informed that there were 2 intervention groups and that the only difference was the way in which skill activities were selected for them by the app.

Once students were given access to a psychological flexibility dashboard, they could practice any activity from that skill set, either by choosing from the list or by allowing the app to select for them. The latter involved a simple randomization process where each activity had an equal chance of selection (1 in 20). Upon completion of an activity, students had the option to complete another within the same psychological flexibility skill set. If they selected “yes,” these steps were repeated until the participant elected to discontinue.

The version of BiSi implemented in this study incorporated medical student feedback from a small feasibility trial of the app [] with the aim of enhancing intervention relevance and engagement. This included clear explanations of what to expect during each stage of the intervention, providing progress indicators for all app components, providing earlier opportunities for personally relevant experiential learning and self-reflection (during stage 1), and delivering content in both written and audio formats where practical (during stage 1). We also introduced positive reinforcement components that were delivered after students completed certain activities or a certain number of activities (eg, achievement badges and experience points). Students were asked to complete a minimum of 4 stage 2 skill activities but were also encouraged to optimize skill learning by practicing more regularly. Students were sent reminders to use the app at 6 PM each day. This frequency and time were based on feedback from the feasibility trial. Although reminders were intended to be delivered using push notifications as per medical students’ reported preferences, this function did not operate as intended during the study, and reminders were sent by email instead.

Outcome MeasuresPsychological Outcome MeasuresOverview

Self-report psychological outcome measures were administered at 2 time points: baseline (collected before randomization) and postintervention measurement (collected 5 weeks after randomization). The outcome measures were presented in the order of the following sections. Participants were required to provide a response to each item before progressing to the next. Owing to the limitations of the app, participants could not alter their responses once submitted.

Burnout

Exhaustion was selected as the primary outcome for this study as it is the most prevalent factor among medical students [] and physicians [] and early improvements in exhaustion following an ACT intervention may prevent the future development of other burnout factors (cynicism) []. Cynicism and academic efficacy were assessed as secondary burnout outcomes. The Maslach Burnout Inventory [] is a valid [], gold-standard [] measure of the 3-factor burnout model. Of the versions available, the General Survey for Students (MBI-GS [S]) [,] was the most appropriate for the medical student cohort. The 16-item self-report questionnaire assesses the degree to which students are experiencing each factor using a 7-point Likert scale ranging from 0 (“never”) to 6 (“every day”). Higher total scores for exhaustion and cynicism and lower total scores for academic efficacy are indicative of higher frequencies of burnout-related experiences. The reliability of the MBI-GS (S) has been demonstrated among medical students [,]. In this study, internal consistency was excellent for exhaustion (Cronbach α=.90) and good for cynicism (Cronbach α=.83) and academic efficacy (Cronbach α=.83). Items were presented in consecutive order, with 1 item displayed per app screen (18 screens, including introductory text).

Well-being

Well-being was measured as a secondary outcome using the Mental Health Continuum–Short Form [], which assesses self-reported hedonic (ie, feeling good) and eudaimonic (ie, functioning well) aspects of well-being []. Participants rate the frequency of 14 well-being experiences during the previous month using a 6-point Likert scale ranging from 0 (“never”) to 5 (“every day”). Total well-being is estimated by summing all scale items. Higher scores reflect higher overall well-being. The scale has demonstrated validity [,], reliability [,], and sensitivity to change in web-based intervention studies []. Internal consistency was excellent in our sample (Cronbach α=.92). Items were presented in consecutive order, with 1 item displayed per app screen (15 screens, including introductory text).

Psychological Flexibility and Inflexibility

The Multidimensional Psychological Flexibility Inventory–Short Form [] was administered to evaluate whether the intervention improved medical students’ psychological flexibility and reduced their psychological inflexibility (secondary outcomes). This 24-item self-report questionnaire assesses the frequency of psychological flexibility and psychological inflexibility experiences during the previous 2 weeks using a 6-point Likert scale ranging from 1 (“never true”) to 6 (“always true”). The scale provides separate average global composite scores for psychological flexibility and inflexibility as recent research suggests that they are related but “conceptually distinct” processes that may have disparate relationships with well-being and psychological distress outcomes and may respond differently to interventions [,,]. The scale has demonstrated validity [,] and reliability [,] and is responsive to changes over time []. This study demonstrated good internal consistency for psychological flexibility (Cronbach α=.87) and inflexibility (Cronbach α=.87). Items were presented in consecutive order, with 1 item displayed per app screen (24 screens).

Psychological Distress (Depression, Anxiety, and Stress)

The Depression, Anxiety, and Stress scale–21 [] is a valid and reliable [,] 21-item self-report questionnaire assessed using a 4-point Likert scale. The measure provides subscale scores that estimate the severity of depression, anxiety, and stress symptoms []. We included this measure to explore whether the ACT intervention improved these secondary outcomes and evaluate whether these psychological distress factors affected engagement in the intervention, as has been observed in previous digital intervention studies []. Internal consistency was excellent for depression (Cronbach α=.90) and good for anxiety (Cronbach α=.81) and stress (Cronbach α=.87) in this study. Items were presented in consecutive order, with 7 items displayed per app screen (3 screens).

Intervention Engagement Outcomes

Behavioral engagement data and subjective intervention feedback were collected from participants throughout the study.

Study Attrition

Attrition was defined as formal withdrawal or loss to follow-up at any stage during the study without completing the postintervention outcome measures.

Intervention Adherence

Adherence to the individualized and nonindividualized intervention arms was defined as the completion of all stage 1 components and engagement in at least 4 skill activities during stage 2. This level of adherence provided students with the opportunity to learn about and practice each of the psychological flexibility processes (stage 1) and practice the skills a few times in their everyday lives (stage 2).

Intervention Feedback

Students were invited to submit feedback on their experience of using the app via a form presented halfway through stage 1. The feedback form was also accessible via the main dashboard. During stage 2, participants rated whether they liked each skill activity they practiced using a single-item binary measure (thumbs up [like] or thumbs down [dislike] icon) []. Participants were invited to report concerns or harms experienced during the study using contact links provided within the app and via email communication (eg, daily reminders).

Data AnalysisPower Analysis

R (statistical computing package; R Foundation for Statistical Computing) was used to calculate the standard power for a 2-tailed independent-sample t test for between-group differences not accounting for repeated measures. This was considered a conservative choice as reliable within-person correlation information needed for power based on a mixed model was not available. This analysis indicated that a sample of 117 participants would provide sufficient power (80%) to detect a clinically meaningful effect size (SD 0.65) between either intervention arm and the control arm in the primary outcome (exhaustion), measured using the MBI-GS (S), with a type-I error rate of 5%. We aimed to recruit up to 153 participants to provide a 30% margin for attrition.

Participant Psychological Characteristics at Baseline

Average participant psychological characteristics at baseline were compared with reference samples (including other medical student or general population samples from previously published studies) using single-sample t tests.

Intervention Effects

This study adopted an intention-to-treat analysis, which included data collected from all participants randomized into a study group. Each participant’s data were analyzed based on the study group to which they were randomized irrespective of their degree of intervention engagement or whether they met the adherence criteria. This approach maintains randomization benefits and minimizes bias when assessing intervention efficacy [].

Differences between the intervention arms were assessed using linear mixed regression models for primary (exhaustion) and secondary (cynicism, academic efficacy, well-being, psychological flexibility and inflexibility, depression, anxiety, and stress) outcomes. A separate model was estimated for each outcome variable. The models included fixed categorical effects for time (baseline as the referent), intervention group, the interaction between intervention and time, and self-reported baseline participant characteristics that were imbalanced after randomization (ie, gender and whether students were studying medicine as their first career [“first career”]). The model included a random participant-specific intercept to account for the repeated measures for each participant. The adjusted difference between the intervention groups in mean change from baseline to postintervention measurement is presented, as well as the within-group change from baseline, 95% CIs, and 2-tailed P values. Model assumptions were assessed by inspecting residual plots. For outcomes that violated modeling assumptions (ie, demonstrated nonnormality of residuals or nonconstant variance), a robust linear mixed-effects model was estimated, with the same fixed and random effects as the previous models.

To control for elevated type-I errors arising from having 2 primary contrasts of interest (each intervention vs control), we followed a hierarchical testing procedure for the primary outcome (exhaustion) where the less intensive intervention (nonindividualized) would only be declared to be significantly different from the control (at a 5% significance threshold) if the more intensive intervention (individualized) was statistically significantly different from the control at a 5% significance level. All other analyses were exploratory. We note that the study was not powered to directly compare the individualized and nonindividualized intervention arms.

Mediation Analyses

We planned to conduct mediation analyses to assess whether changes in process outcomes (psychological flexibility and inflexibility) between baseline and postintervention measurement mediated changes in psychological outcomes. However, the observed intervention effects did not support the implementation of these analyses for reasons outlined in the Results section.

Study Attrition, Intervention Adherence, and Engagement

Baseline participant demographic and psychological characteristics were compared between (1) those who were lost to follow-up at any point during the study versus those who were not and (2) those who met intervention adherence criteria versus those who did not using chi-square tests for categorical variables (or the Fisher exact test where cell size was <5) and independent-sample Student t tests for continuous variables (Welch t tests used where the assumption of equal variance was violated). The average rates of engagement during stage 2 were compared between the individualized and nonindividualized groups using independent-sample t tests.

ResultsParticipant Demographics

A total of 143 medical students were enrolled in this study. shows demographic characteristics of the total enrolled sample and each intervention group. The enrolled participants were aged between 18 and 51 years (mean 24.0, SD 5.48 years), and more than half (88/143, 61.5%) were female. Most participants were nonindigenous (133/143, 93%) and domestic students (135/143, 94.4%) and were training in medicine as their first career (108/143, 75.5%). The average time spent in the workforce was 5.74 (SD 5.71) years. A substantial proportion of the enrolled students (122/143, 85.3%) reported having previously experienced burnout, and 21% (30/143) were engaging in psychological treatment at the time of the study. Students rated the quality of their health, diet, and self-care using a Likert scale ranging from 1 (“very poor”) to 5 (“excellent”). Mean ratings for quality of health (mean 3.72, SD 0.89) and diet (mean 3.61, SD 0.73) fell between “average” (3) and “good” (4), whereas mean ratings for self-care (mean 3.19, SD 0.82) were closer to an “average” rating.

Table 1. Participant demographic characteristics at baseline (by study group allocation and total; N=143).
IIa (n=37)NIb (n=36c)WLd (n=35)Not allocated (n=35)Total enrolledAge (years), mean (SD; range)23.7 (5.60; 18-46)25.4 (5.78; 19-42)22.1 (3.01; 18-31)24.7 (6.55; 19-51)24.0 (5.48; 18-51)Gender, n (%)
Female26 (70.3)17 (47.2)23 (65.7)22 (62.9)88 (61.5)
Male11 (29.7)18 (50)9 (25.7)13 (37.1)51 (35.7)
Nonbinary0 (0)0 (0)3 (8.6)0 (0)3 (2.1)Enrollment, n (%)
Domestic35 (94.6)34 (94.4)32 (91.4)34 (97.1)135 (94.4)
International2 (5.4)1 (2.8)3 (8.6)1 (2.9)7 (4.9)Year of study, n (%)
19 (24.3)6 (16.7)11 (31.4)18 (51.4)44 (30.8)
215 (40.5)14 (38.9)7 (20)7 (20)43 (30.1)
47 (18.9)7 (19.4)8 (22.9)4 (11.4)26 (18.2)
56 (16.2)8 (22.2)9 (25.7)6 (17.1)29 (20.3)Indigenous, n (%)2 (5.4)4 (11.1)0 (0)4 (11.4)10 (7)First career, n (%)26 (70.3)25 (69.4)32 (91.4)25 (71.4)108 (75.5)Years in the workforce, mean (SD)5.19 (6.0)6.54 (5.95)3.91 (3.12)7.34 (6.69)5.74 (5.71)Previous burnout, n (%)34 (91.9)31 (86.1)27 (77.1)30 (85.7)122 (85.3)Current therapy, n (%)6 (16.2)7 (19.4)11 (31.4)6 (17.1)30 (21)Health rating, mean (SD)3.62 (0.92)4.00 (0.80)3.40 (0.98)3.83 (0.86)3.72 (0.89)Diet rating, mean (SD)3.43 (0.87)3.97 (0.57)3.43 (0.71)3.54 (0.82)3.61 (0.73)Self-care rating, mean (SD)3.22 (0.81)3.29 (0.66)3.17 (0.71)3.06 (0.84)3.19 (0.82)

aII: individualized intervention.

bNI: nonindividualized intervention.

cDemographic data missing for 1 participant in this group.

dWL: waiting list.

Participant Psychological Characteristics at Baseline

Outcome scores were calculated for all participants who completed the measures at baseline and only those who were randomized to an intervention arm. Removal of the participants lost to follow-up before random allocation did not greatly alter the mean scores for any psychological outcome. Therefore, we compared the average psychological characteristics of only the participants who proceeded to randomization and were included in the efficacy analyses (108/143, 75.5%) with reference samples (other medical students or the general population) from previously published studies ( [,,,]).

On average, medical students presented a mixed profile of burnout scores, demonstrating exhaustion levels (mean 16.29, SD 7.20) comparable with those of a sample of medical students not experiencing burnout (mean 14.96, SD 5.71; t269=1.69; P=.09), academic efficacy scores (mean 24.69, SD 6.47) comparable with those of medical students who perceived themselves to be high in burnout (mean 24.81, SD 5.35; t216=0.15; P=.88), and cynicism scores (mean 10.87, SD 6.92) significantly lower than those of the burnout sample (mean 14.44, SD 5.59; t216=4.19; P<.001) but higher than those of the nonburnout sample (mean 7.59, SD 5.16; t269=4.46; P<.001) []. Total well-being scores (mean 43.03, SD 13.31) were significantly lower than those of a large multi-institutional sample of medical students in the United States (mean 47.0, SD 12.67; t2796=3.12; P=.002) []. Psychological inflexibility scores (mean 3.21, SD 0.91) were significantly higher than those of a general population sample (mean 2.73, SD 0.90; t2769=5.43; P<.001) []. Average depression (mean 14.89, SD 10.25), anxiety (mean 8.50, SD 8.09), and stress (mean 11.87, SD 10.34) scores for the medical student sample in this study were significantly higher than the general population averages (depression: mean 6.34, SD 6.97, t3020=12.27, and P<.001; anxiety: mean 4.7, SD 4.91, t3020=7.67, and P<.001; stress: mean 10.11, SD 7.91, t3020=2.24, and P=.03) [].

Intervention EffectsOverview

shows the results for the burnout; well-being; psychological flexibility and inflexibility; and depression, anxiety, and stress outcomes.

Table 2. Data summaries of outcomes at each time point (baseline and postintervention measurement) for each treatment arm (mean and SD), mixed model–based estimates of within-group change (95% CI), and estimated between- and within-group change (95% CI and P value).Outcome, subcategory, and intervention groupBaseline (n=108), mean (SD)Postintervention measurement (n=68), mean (SD)Baseline–postintervention measurement estimated change (95% CI)P valueBurnout
Exhaustion (primary)

WLa17.91 (7.02)17.05 (6.81)−1.61 (−3.96 to 0.745)N/Ab

IIc16.65 (7.22)14.48 (6.85)−2.13 (−4.27 to 0.008)N/A

NId14.33 (7.09)14.79 (5.60)−0.01 (−2.51 to 2.49)N/A

II-WLN/AN/A−0.52 (−3.70 to 2.65).75

NI-WLN/AN/A1.60 (−1.84 to 5.03).37

II-NIN/AN/A−2.12 (−5.41 to 1.17).21
Cynicism

WL11.5 (7.35)12.0 (7.62)−0.59 (−1.94 to 0.77)N/A

II11.7 (7.01)9.74 (5.70)−1.76 (−3.91 to 0.39)N/A

NI9.36 (6.32)9.95 (5.54)0.50 (−2.01 to 3.02)N/A

II-WLN/AN/A−1.26 (−4.46 to 1.94).44

NI-WLN/AN/A1.00 (−2.45 to 4.46).57

II-NIN/AN/A−2.26 (−5.57 to 1.05).18
Academic efficacy

WL23.2 (6.29)23.9 (6.27)1.29 (0.51 to 2.08)N/A

II25.9 (6.42)27.5 (5.29)1.25 (0.01 to 2.49)N/A

NI24.9 (6.60)26.4 (6.86)2.33 (0.86 to 3.80)N/A

II-WLN/AN/A0.94 (−0.90 to 2.79).32

NI-WLN/AN/A2.02 (0.02 to 4.03).05

II-NIN/AN/A−1.08 (−3.00 to 0.84).27Well-being
Total well-being

WL39.5 (13.3)40.4 (13.1)3.83 (1.49 to 6.18)N/A

II44.2 (13.7)48.1 (11.3)3.51 (−0.20 to 7.22)N/A

NI45.2 (12.5)52.1 (8.8)7.22 (2.87 to 11.57)N/A

II-WLN/AN/A2.74 (−2.77 to 8.26).33

NI-WLN/AN/A6.46 (0.49 to 12.42).04

II-NIN/AN/A−3.71 (−9.43 to 2.00).21Psychological flexibility
Flexibility

WL3.65 (0.73)3.69 (0.79)0.26 (0.09 to 0.42)N/A

II3.74 (0.87)4.22 (0.83)0.48 (0.22 to 0.74)N/A

NI3.78 (0.73)4.10 (0.72)0.32 (0.02 to 0.62)N/A

II-WLN/AN/A0.50 (0.12 to 0.89).01

NI-WLN/AN/A0.35 (−0.07 to 0.76).11

II-NIN/AN/A0.16 (−0.24 to 0.56).44
Inflexibility