STRENGTHS AND LIMITATIONS OF THIS STUDY

The strength of this study is that it uses a scale with high validity, measured as recommended by the Ministry of Health, Labour and Welfare guidelines, developed by leading medical education experts in Japan.

The limitation of this study is that there may have been some confusion in evaluation using E-POrtfolio of Clinical training (EPOC2) because this study was conducted with residents trained during the COVID-19 pandemic.

Another limitation is that this was the first year that EPOC2 was implemented nationally in Japan, which might have impacted the evaluation process.

Introduction

In recent years, medical education has placed greater emphasis on residents acquiring competencies rather than knowledge alone,1 and there is a need to define and evaluate developmental milestones and entrustable professional activities (EPAs) when assessing whether specific competencies have been acquired during training.2 A portfolio system allows physicians to longitudinally document their experiences and acquired competencies during their training, and such systems are being increasingly adopted in medical education.2

In the context of undergraduate medical education, a survey of the use of portfolios at Liaison Committee on Medical Education-accredited medical schools in the USA reveals that just less than half (47%) of the 71 schools responding to the survey used portfolios and that the majority of those used electronic portfolios (e-portfolios).2 A report from University Medical Center Utrecht in the Netherlands describes how an e-portfolio system in the clinical medicine curriculum allows students to efficiently record observations and feedback during clinical practice as well as to track their progress.3 Other reports on the introduction of e-portfolios include those from Maastricht University in the Netherlands,4 Virgen Macarena University Hospital in Spain,5 Catholic Kwandong University in Korea,6 McGill University in Canada7 and National Taiwan University in Taiwan.8 It should be noted that none of these are national-level implementations of the e-portfolio system. One report from Switzerland discusses the potential importance that an e-portfolio system could have, given the country’s recent national-level introduction of its competency and EPA-based approach to medical education.9

In the context of postgraduate medical education, one systematic review reports that the use of portfolios will allow for a multifaceted and unified understanding of each learner’s progress.10 Portfolios may also be beneficial in enhancing individual responsibility for learning and helping with professional development.11 The electronic format (ie, e-portfolio) could offer additional advantages and promote more enthusiastic use.11 One report on the large-scale use of e-portfolios in postgraduate medical education comes from Canada, where an e-portfolio system was designed by the Royal College of Physicians and Surgeons of Canada to support continuing professional development in Canada.12 However, to our knowledge, there were no reports of e-portfolios being used uniformly at the national level in the first and second year of initial residency after graduation from the medical schools.

Using longitudinal evaluation data from an e-portfolio system, researchers and educators can examine learning trajectories to measure learners’ developmental progress, that is, growth and acquisition of competencies over time. A recent study used longitudinal cohort data of family medicine residents from entry to graduation to investigate the learning trajectories and patterns of the residents,13 while we have used group-based trajectory modelling14–18 to categorise the learning trajectories of medical and dental students in their undergraduate education into different groups and to identify factors that determine the probability that an individual will follow one of these trajectories.19 20

EPOC2 (E-POrtfolio of Clinical training) is an e-portfolio system developed in Japan that covers the entire clinical clerkship to residency period. In Japan, the Ministry of Health, Labour and Welfare defines the goals that junior residents should achieve within 2 years of training across three domains: (1) fundamental values as physicians (Professionalism), (2) specific competencies and capabilities required of physicians and (3) basic clinical tasks that can be performed almost independently (Independent Practice Allowed in Conditional Situations). EPOC2 facilitates on-the-spot input of evaluations for these three domains. Currently, EPOC2 is used by about 800 clinical training hospitals and more than 8000 junior residents, allowing researchers to examine data on junior residents nationwide. The objective of this study is to use the EPOC2 data to identify the learning trajectory of junior residents in order to provide insights into the provision of better postgraduate and undergraduate medical education in Japan.

MethodSample

A seamless ICT-based undergraduate and postgraduate evaluation system (EPOC2) has been used to evaluate clinical training at training hospitals nationwide, and the system is operated by the University Hospital Medical Information Network (UMIN) Center, The University of Tokyo Hospital. Data obtained for training evaluations using EPOC2 are anonymised at the UMIN Center and provided to the Tokyo Medical and Dental University for secondary analysis in this longitudinal cohort study. Included in the study were junior residents who participated in clinical training programmes at clinical training hospitals nationwide between April 2020 and March 2022. Data were initially collected for 8721 residents. Some students were simultaneously enrolled in two departments (referred to as ‘parallel training’), resulting in duplicate datasets for the same period that could not be consolidated for analysis. Additionally, some residents’ records lacked complete information across all items. After excluding these cases, data from 7671 residents were analysed in this study (figure 1).

Figure 1

Flow diagram of study participants.

E-POrtfolio of Clinical training

The Model Core Curriculum for Medical Education outlines the goals of medical education in Japan regarding essential practical medical skills (knowledge, skills and attitudes) that medical students should acquire by the time of graduation, and states ‘the importance of seamless perspectives of education, learning and training ranging from undergraduate to postgraduate education’.21 Under these circumstances, the EPOC Steering Committee of the National University Hospital Council of Japan developed EPOC2 with the cooperation of the UMIN Center.22 EPOC2 allows on-the-spot input of evaluations (self-evaluations and supervisor/senior physician evaluations) using a smartphone. It complies with the Ministry of Health, Labour and Welfare’s guidelines for clinical training guidance for physicians.23 24 EPOC2 has been in operation nationwide since fiscal year 2020. Approximately, 90% (over 8000) of all residents in Japan are using EPOC2 for evaluation in their clinical training.25

Clinical training and evaluation in junior residency in Japan

After 6 years of medical school, medical students take the national board examination and, if they pass, receive training in each field of medicine for a certain period of time (called block rotation) during their 2-year training period as a junior resident.23 Internal medicine, surgery, paediatrics, obstetrics and gynaecology, psychiatry, emergency medicine and community medicine are positioned as compulsory fields. In addition, training at a general ambulatory site is also compulsory. As a general rule, the duration of training is at least 24 weeks in internal medicine, 12 weeks in emergency medicine and 4 weeks each in surgery, paediatrics, obstetrics and gynaecology, psychiatry and community medicine. For training at a general ambulatory site, at least 4 weeks of training in block rotation or parallel training is required. Parallel training is defined as training in emergency or general outpatient care at the same time as training in other areas of the programme. There are three objectives in the training programme: (A) ‘Fundamental Values as Physicians (Professionalism)’, (B) ‘Competencies and Capabilities’ and (C) ‘Independent Practice Allowed in Conditional Situations’. Each of these categories consists of more detailed subitems (table 1). The junior resident evaluation for clinical training consists of: (1) evaluation during the training period (formative evaluation) and (2) evaluation at the end of the training period (summative evaluation). Formative evaluations are assessed using the resident evaluation form by the junior residents (self-evaluation) and supervisors, senior physicians and other healthcare professionals at the end of each rotation in each clinical department. This evaluation can be done electronically using EPOC2. The current study used data from the formative evaluation. The resident evaluation form consists of 1, 2 and 3, which correspond to the evaluation of the attainment goals A, B and C, respectively, as defined above. Similarly to the above, each of these categories consists of more detailed subitems (table 1). Each was evaluated using a rubric system, ranging from level 1 to level 4. Junior residents are expected to reach level 3 in all items.

Table 1

Resident evaluation form

Statistical analysis

Group-based trajectory modelling was used to classify the trajectories of the evaluated scores over the 2 years of residency into different characteristic trajectory groups, as described previously.14–18 Briefly, the number of trajectory groups and trajectory shapes were determined by maximising the Bayesian information criterion (BIC).18 That is, first, all trajectories were set to be quadratic, the number of trajectory groups was increased from two to six for comparison and the number of trajectory groups was chosen between two and six so as to maximise the BIC for each item in each evaluation form. After determining the number of trajectories, the shapes of each trajectory were compared by changing the shape of each trajectory to 0 (parallel to the x-axis), 1 (linear function) and 2 (quadratic function), and the shape of each trajectory that maximised the total BIC for each item in each evaluation form was selected. All statistical analyses were conducted using Stata MP statistical package, V.17 (StataCorp, College Station, Texas, USA) with the ‘traj’ plugin for group-based trajectory modelling.16

Results

Table 2 shows the demographic characteristics of study participants. The mean age of the junior residents was 25 years, with approximately 57% identifying as male and 29% as female. About 38% of the junior residents were trained at university hospitals.

Table 2

Demographic characteristics of study participants (n=7671)

Figure 2 shows the trajectory of evaluation based on evaluation forms A1–A4. Regarding self-evaluation, A-1 (figure 2a, accountability), A-2 (figure 2b, altruistic attitude) and A-4 (figure 2d, lifelong learning) showed a similar trajectory and were divided into six groups. Interestingly, the pattern of the self-evaluation of A-3 (figure 2c, respect for humanity) was slightly different from the pattern of the rest of the evaluation forms, with no group starting at 2 (below expectations) and reaching 3 (meets expectations). The pattern was otherwise similar to that of the other evaluation forms, notwithstanding the slightly different appearance of the group that started at a little above 2 and failed to reach 3 at the end (about 4% of them).

Figure 2

Trajectory of evaluation based on evaluation forms A1–A4.

In the evaluation by the supervisor/senior physician, the patterns were similar across the four evaluation forms. However, the percentage of the lowest rated group (starting below 3 and reaching 3 at the end) was different, ranging from 1% to 2% in A-1 (figure 2e, accountability), A-2 (figure 2f, altruistic attitude) and A-3 (figure 2g, respect for humanity), whereas in A-4 (figure 2h, lifelong learning), the percentage was 4.7%.

Online supplemental figure 1 shows the trajectory of evaluation based on evaluation forms B1–B4. Regarding self-assessment, B-1 (online supplemental figure 1a, ethics in medicine and medical practice), B-3 (online supplemental figure 1c, procedural skills and patient care) and B-4 (online supplemental file 1d, interpersonal and communication skills) showed a similar trajectory. Interestingly, the pattern of the self-assessment of B-2 (online supplemental figure 1b, medical knowledge and problem-solving) was slightly different from the pattern of the rest of the evaluation forms, with no group starting near a value of 3 (expected level at the end of clinical training) and growing to approach 4 (expected level as a senior doctor). Furthermore, the percentage of groups that remained at 3 from the beginning was also smaller in B-2 (27.0%) compared with the other forms: 33.5% in B-1, 31.5% in B-3 and 33.5% in B-4.

In the evaluation by the supervisor/senior physician, B-1 (online supplemental figure 1e, ethics in medicine and medical practice), B-2 (online supplemental figure 1f, medical knowledge and problem-solving) and B-4 (online supplemental figure 1h, interpersonal and communication skills) showed similar patterns. Interestingly, the pattern of B-3 (online supplemental figure 1g, procedural skills and patient care) was slightly different from that of the rest of the evaluation forms, with no group that starting slightly above 3 and slowly rising; instead, another group was observed that started between 1 (expected level at the beginning of clinical training) and 2 (expected level at the midpoint of clinical training) and did not reach 3. The pattern was otherwise similar to that of the other evaluation forms.

Online supplemental figure 2 shows the trajectory of evaluation based on evaluation forms B5–B9 (online supplemental figure 2a–e). Regarding self-assessment, in addition to the three groups that remained near or above the value of 3 from the beginning to the end, there was one group that increased from a low value to reach the value of 3 as training progressed and there were two groups that increased from a low value but did not reach 3.

In the evaluation by supervisor/senior physician, B-5 (online supplemental figure 2f, practice in interprofessional teams) and B-9 (online supplemental figure 2j, lifelong learning) showed similar patterns. Regarding B-7 (online supplemental figure 2h, medical practice in the context of society) and B-8 (online supplemental figure 2i, scientific exploration), the pattern is almost identical to that of B-5 and B-9, the only difference being that the starting score for the group that rises to 3–4 at the midpoint and settles at 3 is lower and closer to 1. Regarding B-6 (online supplemental figure 2g, patient safety and quality of medical care), compared with the patterns observed in other evaluation forms, the group that started above 3 and rose slowly was absent; instead, a group that started low and did not reach 3 was observed. Also, the starting score of the group that rose to 3 or 4 at the midpoint and settled at the end was closer to 3 and ended higher than 3.

Online supplemental figure 3 shows the trajectory of evaluation based on evaluation forms C1–C4 (online supplemental figure3a–d). Regarding self-assessment, the major difference compared with the A and B evaluation forms is that only the self-assessment for C1–C4 showed a group that remained at or below 2 (able to perform under circumstances where a supervising physician can respond immediately) until the end, with this group accounting for 2.8% to 7.0%.

In the evaluation by supervisor/senior physician, C-1 (online supplemental figure 3e, general ambulatory care), C-2 (online supplemental figure 3f, ward care) and C-3 (online supplemental figure 3g, primary emergency care) showed similar patterns. All groups started at a level below 3 (almost entirely able to perform by themselves), and some groups reached a score higher than 2 but below 3, some groups approached 3, and one group rose sharply in the second half of the first year, ending up between 3 and 4 (a level to mentor younger doctors). Regarding C-4, (online supplemental figure 3h, community-based healthcare), there was a group starting near 4 at the beginning and ending near 3, but this accounted for only a small percentage of the total residents.

Discussion

Using EPOC2 data collected throughout Japan and group-based trajectory modelling, we found that growth trajectories based on resident self-evaluations and supervisor/senior physician evaluations fall into six groups whose characteristics vary depending on the content of the evaluation forms. Regarding evaluation based on evaluation forms A1–A4, the pattern of the self-evaluation of A-3 (respect for humanity) was slightly different from the pattern of the rest of the evaluation forms (A1: accountability; A2: altruistic attitude; A4: lifelong learning). Specifically, the group that started at 2 and reached 3, which was present in other forms, disappeared. This may be because respect for humanity is a necessary quality for all human beings, not just physicians, while the other items are nurtured during training in undergraduate and postgraduate education. In addition, a notable difference in the supervisor/senior physician evaluations is that the number of participants in the group with the lowest scores in A-4 (lifelong learning) was higher than in the other forms. This result suggests the need to further strengthen education on attitude toward self-development in undergraduate and postgraduate education.

In the trajectories of scores for Evaluation Form A, there are no groups of residents whose trajectories begin near a score of 1. It is important to note that the meaning of a score of 1 differs across evaluation forms: in Evaluation Form A, it represents ‘Significantly Below Expectation’; in Evaluation Form B, it corresponds to ‘Level Expected at the Commencement of Residency Training’; and in Evaluation Form C, it denotes ‘Able to Perform Under the Supervision of the Medical Advisor’. Consequently, caution should be exercised when making direct comparisons between forms. The absence of groups of residents starting near a score of 1 in the trajectories for Evaluation Form A might be attributed to the nature of the items assessed. Evaluation Form A focuses on ‘Fundamental Values as a Physician’. Residents who have successfully graduated from medical school and entered junior residency are likely to have already developed a baseline level of these fundamental values. This could explain why no groups start near a score of 1 in Evaluation Form A.

Regarding assessment based on evaluation forms B1–B4, the pattern of the self-assessment of B-2 (medical knowledge and problem-solving) was slightly different from the pattern of the rest of the evaluation forms (B1: ethics in medicine and medical practice; B3: procedural skills and patient care; B4: interpersonal and communication skills). Specifically, in B-2, the group that started near a value of 3 and grew to approach 4 disappeared. Additionally, the percentage of groups that remained at 3 from the beginning was also smaller in B-2 (27.0%) compared with the other forms: 33.5% in B-1, 31.5% in B-3 and 33.5% in B-4. This suggests that residents' confidence at the start of their residency is relatively low, especially in medical knowledge and problem-solving skills, and residents do not necessarily believe that they have sufficient medical knowledge and problem-solving skills during their residency. Since confidence has been shown to be critical to a resident’s ability to lead a team and gain trust from other team members,26 27 these findings suggests that there is a need to help graduating medical students and residents build confidence that is appropriate to their stage of training, as well as to their actual knowledge, skills and behaviours.

A notable difference between B1 and B4 in the supervisor/senior physician evaluations was that in B-3 (procedural skills and patient care), compared with the other groups, there was no group that started slightly above 3 and gradually increased, but instead a group that started between 1 and 2 and did not reach 3 was observed. The reason for this result may be that, compared with the items evaluated in the other forms, many of the medical and patient care skills can be mastered only during and after residency, even though they can be learnt to some extent as a medical student. As more medical practice is expected to become feasible for medical students with the enactment of the recent change in the law in 2023,28 future studies evaluating the impact of those policy changes are warranted.

Regarding assessment based on evaluation forms B5–B9, while there was no difference in the pattern of self-assessment, there was a notable difference in the supervisor/senior physician evaluations in B-6 (patient safety and quality of medical care). That is, compared with the other forms (B-5: practice in interprofessional teams; B-7: medical practice in the context of society; B-8: scientific exploration; B-9: lifelong learning), a group that started at 3 or higher and rose slowly was eliminated; instead, a group that started low and never reached 3 was observed. This suggests the importance of strengthening education on the management of quality and safety in healthcare, which is consistent with previous studies.29 30 Since residents practice medicine under the guidance of their supervisors, further research is needed to determine how to systematically teach them the importance of quality and safety in healthcare.

Regarding evaluation based on evaluation forms C1–C4 (C-1: general ambulatory care; C-2: ward care; C-3: primary emergency care; C-4: community-based healthcare), the major difference compared with the A and B evaluation forms is that only the self-evaluation for C1–C4 showed a group remaining at or below 2 until the end. Since all the supervisor/senior physician evaluations for C1–C4 reached 2 or above, this may indicate that the residents’ self-confidence is not sufficient even at the end of the residency. This is similar to the results of B2 (medical knowledge and problem-solving), where the residents were less confident in their medical knowledge and problem-solving skills. It is true that residents sometimes express concern about their ability to perform medical procedures independently.31 However, a previous study showed that independent decision-making experience and good back-up support from senior residents were associated with increased resident confidence, whereas high patient load, long work hours and abusive interactions were associated with decreased confidence.32 Future research is needed to determine how to appropriately increase resident confidence in the context of postgraduate medical education in Japan.

There are several limitations to this study. First, there may have been some confusion in evaluation using EPOC2 because this study was conducted with residents trained during the COVID-19 pandemic and this was the first year that EPOC2 was beginning to be used nationally in Japan. Second, because evaluations, both self-evaluations and supervisor/senior physician evaluations, are subjective, further research is needed to determine the validity of the items. However, there are efforts to ensure the validity of supervisor/senior physician evaluations by having multiple supervisor/senior physician evaluations, and also by incorporating evaluations from patients and from non-physician medical staff. Moreover, the items measured by EPOC2 are in accordance with the Japanese Ministry of Health, Labour and Welfare’s guidelines for clinical training guidance for physicians,23 and their validity is to some extent assured by experts in medical education. Third, the BIC evaluates model fit, with additional trajectory groups improving the BIC (model fit) only when the improvement in log-likelihood outweighs the penalty for adding parameters.14 Due to this penalty, the BIC criterion tends to prefer models with fewer groups.14 In this analysis, however, the model with six groups provided the best fit for all items, reflecting the diversity in residents' learning trajectories, which is intriguing. Although we tested a range of trajectory groups from 2 to 6 and found that six groups provided the best fit within that range, expanding the range further might yield a different optimal number of groups. Nevertheless, adding more groups would complicate interpretation, so we chose to use six groups for this study. Finally, qualitative descriptors were converted to numerical values, and trajectory analysis was conducted. It is worth noting that this evaluation system includes intermediary scores (eg, 1.5, 2.5 and 3.5), making it somewhat different from a traditional Likert scale. However, future research are warranted to consider alternative analysis methods that preserve the original scale to validate these results.

EPOC2 is used by approximately 800 clinical training hospitals and more than 8000 junior residents throughout Japan. The strength of this study is that we were able to analyse representative data from junior residents throughout Japan.

This study has the following implications. First, since residents appear to lack confidence in their medical knowledge, problem-solving skills and ability to independently see patients, it is important to help graduating medical students and residents build confidence that aligns with their actual stage of training, knowledge, skills and behaviours. Providing opportunities for independent decision-making with reliable backup support, as noted in a previous study,32 may be beneficial. However, it is crucial to remember that confidence does not always equate to capability. Therefore, training should aim to foster confidence that genuinely reflects each resident’s level of knowledge, skills and behaviours appropriate to their training stage. Second, our results also suggest the need to further strengthen education on attitude toward self-development and the management of quality and safety in healthcare in undergraduate and postgraduate education, which is consistent with previous studies conducted in other countries.29 30 33 34

In conclusion, using data from EPOC2 and group-based trajectory modelling, we found that growth trajectories based on resident self-evaluations and supervisor/senior physician evaluations fall into six groups, each with its own distinct characteristics depending on the content of the evaluation forms. The results of this study have the potential to provide valuable insights into the developmental trajectory of junior residents and to identify opportunities for enhancing their training programmes in Japan. They also highlight the value of a nationwide evaluation system, allowing researchers to analyse the outcomes of clinical education in a country using uniform indicators.

Data availability statement

De-identified data used in this study may be shared upon reasonable written request to the corresponding author(s) and the committee that manages the data. Access to de-identified data requires procedures and institutional review boards as directed by the committee and will be reviewed by the committee that manages the data.

Ethics statementsPatient consent for publication

Not applicable.

Acknowledgments

We would like to thank Shuichi Tanae at UMIN Center for processing and providing the data for secondary analysis. This study was funded by Health Labour Sciences Research Grant (202103013A). The funders had no role in the study design, data collection and analysis, decision to publish or preparation of the manuscript.