Introduction

The insulin pump (PUMP) and the continuous glucose monitoring system (CGMS) are essential advancements in modern diabetes mellitus (DM) care, designed to optimize blood glucose regulation. The PUMP delivers precise doses of insulin, mimicking natural insulin release, while the CGMS provides continuously real-time data on glucose levels, enabling timely adjustments. Together, these technologies empower patients with diabetes to maintain stable glucose control effectively, each contributing uniquely to comprehensive diabetes management.1

The PUMP continuously infuses insulin to regulate blood glucose by mimicking natural insulin secretion. It delivers both basal and bolus doses, helping maintain stable glucose levels and reducing the discomfort of frequent injections. Key components include the pump unit, insulin reservoir, and tubing with an infusion set. The insulin reservoir stores the insulin, while the pump unit features a display and controls for setting basal rates and pre-meal doses. This system minimizes hypoglycemia and reduces the discomfort of multiple injections by ensuring stable, continuous insulin delivery.

The CGMS is a minimally invasive technology designed to track blood glucose levels in real time. It uses microelectrodes embedded in the subcutaneous tissue to detect electrical signals from glucose oxidation in the interstitial fluid, providing an indirect measurement of glucose levels. CGMS continuously monitors glucose, enabling the detection of hypoglycemia and hyperglycemia, offering timely alerts, and helping patients understand their glucose fluctuation patterns. Together, the insulin pump and CGMS offer a comprehensive solution for managing diabetes, ensuring more stable and consistent control throughout the day.

Training medical students in the use of PUMP and CGMS requires a well-rounded approach, combining technical skills with empathy and patient communication. A comprehensive evaluation tool should focus on 1) Medical Knowledge: Students must grasp diabetes management principles, including indications, contraindications, and complications, and apply this knowledge to clinical scenarios. 2) Operational Skills: Students should be proficient in operating and troubleshooting insulin pumps and CGMS, ensuring effective diabetes care. 3) Humanistic Care: This assesses students’ communication, empathy, and professionalism, observed through interactions with patients and role-playing scenarios. 4) Evaluation and Feedback: Real-time observation of students’ skills enables immediate, constructive feedback, helping students refine their abilities and boost their confidence in clinical practice. This approach ensures a balanced development of both technical and interpersonal skills in diabetes care.2,3

Skill training in the medical field faces several challenges, particularly the gap between theoretical knowledge and practical application. While medical students may have strong theoretical foundations, they often lack hands-on experience, especially with complex tools like insulin pumps and CGMS. Moreover, such procedures require both technical expertise and strong interpersonal skills, including effective patient communication and empathy. Limited time and resources in clinical settings further complicate training. To address these challenges, structured programs that combine technical skills with humanistic care, along with consistent feedback, are essential for effective learning and improvement.

Several methods have been explored to enhance medical student training in DM management. Research indicates that hands-on experience with diabetes management tools improves students’ understanding of the condition and enhances their ability to educate patients. These approaches offer valuable opportunities for students to develop both their clinical skills and their communication abilities, ensuring a more comprehensive training experience for effective diabetes care. Training tools include virtual simulations,4 simulation-based learning,5 and case-based learning,6 which provide risk-free environments for practicing and patient communication.

Direct Observation of Procedural Skills (DOPS) is an assessment method introduced in the 1990s to evaluate medical students’ clinical competencies. By observing students perform specific procedures, such as using medical devices or administering treatments, educators assess both technical skills and communication abilities in real-time. DOPS provides immediate feedback to improve student performance, making it a valuable tool in medical training for ensuring readiness in practical tasks. DOPS has been implemented across various medical disciplines, with significant research conducted on its effectiveness in improving clinical skills in fields such as surgery,7 and other procedure training.8,9

While it helps refine technical skills and improve clinical performance, assessing both competence and confidences, traditional DOPS has limitations, such as increasing student stress, evaluator bias, and logistical challenges in busy clinical settings.10 To address these challenges, improvements have been made in recent years to expand its advantages emphasizing real-time, direct observation and objective feedback. By prioritizing immediate evaluation and constructive comments, students can refine their techniques and develop better clinical proficiency in a more efficient manner.7,9,11 In this study, the DOPS was modified to include the PUMP and CGMS assessments of medical students in the Department of Endocrinology.

MethodsParticipants and Setting

Two DOPS assessments were administered before and after training to all 64 residents in the Department of Endocrinology at Pudong Hospital, affiliated with Fudan University, during the period from June 2023 to June 2024. During the initial DOPS assessment, the trainees were informed of these assessments as part of their rotations. Under the supervision of their rotation tutors, the students were guided to pre-designated patients with appropriate indications to perform the PUMP and CGM procedures. Their performance was then evaluated, and feedback was provided by the tutors. The modified DOPS was used as a self-evaluation tool during the rotation period preceding the second assessment. After the initial evaluation and feedback from their rotation tutors on task performance, the students were encouraged to reflect on their results and practice the tasks voluntarily during their daily activities, preparing for the second assessment.

Instrument Development

The instrument, developed based on international DOPS standard, incorporates the specific characteristics of diabetic patients, as well as the protocols and scoring systems for PUMP and CGMS procedures. It assesses students’ competencies in diabetes management, evaluating both clinical knowledge and technical skills. A bilingual Chinese-English version is also provided. The current questionnaire has been preliminarily tested on small samples, demonstrating effectiveness. The following five domains were included in each DOPS form of PUMP and CGMS assessment: 1) indications for wearing an insulin PUMP and continuous glucose monitoring and execution time; 2) communication skills, which evaluate the students’ capacity to communicate with and obtain essential patient information, as well as their empathy and humanism. Communication time will also be included as an assessment criterion. 3) Skill proficiency: This item evaluates the student’s actual operative ability in the insertion of a PUMP and CGMS and fundamental medical concepts, such as the requirement for sterility and precautions for MRI when wearing PUMP and CGMS. 4) Ability to seek assistance: This item assesses a student’s collaborative abilities when faced with a challenge. 5) Overall performance: The supervisor provides a comprehensive and general assessment of the score based on the student’s overall performance on the examination, which is indicative of the student’s clinical competence. Tables 1 and 2 show the five domains that comprise the DOPS examination for the PUMP and CGMS, respectively.

Table 1 The Modified DOPS-PUMP Form for Assessment of Trainees Interviewed with the Patients with Diabetes Mellitus (DM)

Table 2 The Modified DOPS-CGMS Form for Assessment of Trainees Interviewed with the Diabetes Mellitus (DM) Patients

The Setting of the DOPS-PUMP and DOPS-CGMS Examination

Two DOPS examinations were conducted at the start and end of rotation. Students will be provided with information regarding the DOPS examination form and encouraged to consult with their supervisors regarding the appropriate warnings and indications for the use of the CGMS and PUMP. The teachers who supervised the two examinations were identical; however, the patients were distinct and had varying disease conditions. The student completed the entire process of installing the insulin PUMP and CGMS, communicated with the patients, and conducted an interview with the patients at the bedside during the DOPS examination.

The supervisor evaluated the performance of the students and the procedure time during the interviews and installation. The student receives necessary feedback from the supervisor during the operation. Students will be informed of the performance and disadvantages of each item in their initial examination, as well as the field that requires improvement following the initial examination. Participants were able to engage in active practice after the initial examination. At the end of his rotation in the Department of Endocrinology, the student will be administered an additional DOPS examination, and the results will be recorded in their rotation record. Table 3 shows the details of each patient who was interviewed.

Table 3 The Characteristics of the Patients Involved in This DOPS-PUMP/-CGMS Study

Statistical Analyses

All statistical analyses were performed using SPSS (version 25, USA) and Prism (GraphPad, version 10.0) software. The Kolmogorov–Smirnov test was used to assess the normal distribution of the data. The five domains, including score and time, using the DOPS-PUMP/CGMS scale were compared using a parametric or non-parametric method-based paired test. Spearman correlation coefficient was used to examine the interrelationships between each item. Multilinear regression analyses were performed to determine the critical traits that contributed to the overall performance score, assessed using DOPS. Statistical significance was set at p < 0.05.

ResultThe Comparisons on the Progress of Scores and Time When Assessing 5 Domains of DOPS

The DOPS forms were disseminated to all 64 students before the examination, which lasted approximately 1 month. The response rate was 100%. As anticipated, the scores for each domain on the DOPS form substantially increased when the evaluation of PUMP/CGMS indication, communication skills, PUMP/CGMS skill proficiency, seeking assistance, overall performance scores, and time use was conducted (p<0.0001) (Figures 1 and 2).

Figure 1 Increased scores on the DOPS in 5 domains including PUMP indication evaluation (A), communication skill (B), PUMP skill proficiency (C), Seek for assist (G), and overall performance (H), with improvement on PUMP evaluation time (D), communication time (E), process time (F), and completion time (I) after DOPS training.

Note: ****p<0.0001.

Figure 2 Increased scores on the DOPS in 5 domains including CGMS indication evaluation (A), communication skill (B), CGMS skill proficiency (C), Seek for assist (G), and overall performance (H), with improvement on CGMS evaluation time (D), communication time (E), process time (F), and completion time (I) after DOPS training.

Note: ****p<0.0001.

The Factors Associated with the 5 Domains Scores Before and After Training in PUMP and CGMS

The Spearman correlation analyses on the five domains of the PUMP evaluation revealed that the initial PUMP indication evaluation score was significantly associated with the scores of the other four domains and negatively correlated with the time used in indication evaluation, skill processing, and completion time of the whole evaluation. The initial communication skill score was significantly correlated with the performance of seeking assistance and played a significant role in overall performance and time use, as well as influencing the time used to process the operation of PUMP and CGMS. The performance in seeking assistance also influenced the time used in the completion time, in addition to the overall performance score, which skill proficiency score also made a significant contribution. Intriguingly, the outcome of the initial assessment was associated with the time used in the final evaluation, particularly PUMP indication evaluation performance.

In the re-examination at the end of the rotation, the performance of communication skills was associated with the performance of PUMP skill proficiency, whereas seeking the fore-assist score was significantly negatively related to communication time. In addition, the time used in the communication and processing evaluation was significantly associated with the completion time of the entire re-examination. The details of the DOPS-PUMP evaluation correlation analyses are presented in Tables 4 and 5.

Table 4 The Correlation Between 5 Domains Scores and Other Elements Before Training in DOPS-PUMP Evaluation

Table 5 The Correlational Relationship Between 5 Domains Scores and Other Elements After Training in DOPS-PUMP Evaluation

In contrast, in two turns of the CGMS examinations, distinct interrelationships existed within the DOPS-PUMP study. While the performance score for the other four domains was only positively correlated with the overall performance score, the communication skill evaluation was significantly related to time use, and the completion time had a significantly negative relationship with seeking assistance. In a reexamination session, a substantial correlation was observed between the overall performance score and the other four domains. In particular, the reexamination score of the CGMS indication was positively correlated with time in the initial evaluation of the CGMS indication (Table 6).

Table 6 The Correlational Relationship Between 5 Domains and Other Elements Before and After Training in DOPS-CGMS Evaluation

The Multilinear Regression Analyses on the Major Components Contributed to the Progress on the Increased Overall Score

Thereafter, in multilinear regression analyses for each scale, include both the performance score and time used in each scale, and the details of the patients included in this study. Interestingly, despite the significant contribution from the scale of performance score and time use, whereas the patient information of this study was excluded from the calculation, dissimilar results with the CGMS evaluation were found; the R square for PUMP did not completely contribute to the scores and time use of the five domains. Details of the multilinear regression are presented in Tables 7 and 8.

Table 7 Multiple Linear Regression Analysis of Factors Influencing the Overall Performance Score Before and After Training (DOPS-PUMP)

Table 8 Multiple Linear Regression Analysis of Factors Influencing the Overall Performance Score Before and After Training (DOPS-CGMS)

Discussion

The DOPS examination is a novel educational tool that is beneficial for evaluating the resourcefulness and operative ability of medical students in real-world healthcare environments.12 It may also enable students to immediately address discrete non-specialty problems in a clinical practice setting by applying their theoretical learning, intuition, and ability to deal with them. In this instrument major, students were assessed for their capacity for reflective thinking, communication, independent reasoning, operations, cooperation, and comprehension. Despite the incorporation of numerous innovative teaching paradigms in the clinical setting for rotation students, such as the Mini-CEX, which was implemented in previous investigated rotation residents,13 the assessment of endocrinology operations has not received significant attention. Consequently, in this pilot study, the DOPS scale was implemented to evaluate students’ skills in the implementation of insulin PUMP and CGMS devices, which are two essential and frequently utilized devices for the management of blood glucose in patients with diabetes (Figure 3).

Figure 3 The illustration concisely displays the processing of installing of an insulin PUMP in a diabetic patient. The components of insulin PUMP are annotated in the illustration.

In the initial section of the self-contrast analysis, the performance scores and time use of the students in the five domains were found to experience substantial improvement following training following the initial examination. This indicates that feedback on their performance had a noticeable beneficial effect on all aspects of a student’s DOPS ability, thereby demonstrating the effectiveness of the DOPS paradigm in enhancing students’ operative skills. In subsequent correlational analyses, the results of PUMP and CGMS demonstrated significant disparities in the interrelationships between performance scores and time use. The items of evaluation of indications for PUMP/CGMS and communication, as well as seeking assistance, played a significant role. The supervisor’s feedback during the evaluation time significantly influenced performance in the final re-examination, indicating that the supervisor’s feedback was essential. This phenomenon has been documented and illustrated in numerous prior investigations.14–16

An intriguing interrelationship existed between the five domains, their scores, and time use. For example, the performance of the evaluation of insulin PUMP indications was significantly associated with process skill proficiency and seeking assistance. Additionally, there was a significant relationship between communication skills and seeking assistance scores, indicating that students’ understanding of the mechanism of PUMP and independent thinking was intimately associated with PUMP operation proficiency. However, the cooperative characteristics of medical students are essential to their everyday duties.17 Additionally, their excellent communication abilities may help them effectively seek assistance from other individuals or professionals to achieve their goals.

These five domains of DOPS are particularly significant in contemporary medicine as there is an increasing demand for medical professionals to participate in active collaboration and communication.18 Excellent communication traits or language expression abilities may help them to smoothly and effectively seek assistance from other individuals, professionals, or teams to achieve their appeals.19 Another benefit for a medical student in DOPS evaluation to enhance their communication skills is to lessen the mutual impedance of the doctor and the patients, promote the smoothness and proficiency of the subsequent process, where basic ability, artistry communication skills, and, more importantly, a doctor’s empathy play a significant role.20–22 DOPS examinations provide medical students with the opportunity to enhance their communication and collaborative skills. The present research demonstrated a distinct positive correlation between skill proficiency and communication time. In particular, the student was able to acquire improved knowledge of the patient’s condition, and the mutual impedance between the doctor and patient was reduced as a result of the comprehensive interviews. The overall completion time was also influenced by the operation proficiency of the PUMP or CGMS, which suggests that medical students’ hands-on ability is essential for clinical operations.

The results of regression analyses virtually alert us to the discrepancy in PUMP and CGMS evaluation that the proportion of an individual’s inner property may be accountable to increase, accompanied by an increase in the degree of complication and difficulty of their current operating task. The regression calculation excluded the possibilities of the implications from the patient’s selection and background information (Table 3). This surprising consequence advised us that the evaluation of implications from the student’s traits or self-feedback for the operation-related assessments could be utilized as an additional evaluation project, and enhancement of a student’s mental strength based on the variation scores assessed by corresponding students will help them accomplish medical-related problems in their daily work.23

Limitation

The evaluation of self-reflection or interview skills can be improved by further refining the scale into the forms of this study. Furthermore, the scale should incorporate feedback content and timeframes, as well as the recommendations of trainers. However, it is essential to incorporate the new practical instrument into future research to assess the various characteristics of students and subsequently aid them in the completion of their clinical duties.

Conclusion

In the current pilot study, utilized modified DOPS methods in the field of endocrinology can assist students in enhancing their hands-on skills in the management of patient blood glucose using PUMP and CGMS. This investigation demonstrated that the DOPS is a practical and comprehensive instrument for assessing the clinical qualities of students in the endocrinology department, as well as for facilitating the enhancement of their clinical skills.

Data Sharing Statement

All data generated or analyzed during this study are included in this published article.

Ethical Statement

The study, including the application of PUMP, CGMS treatments, and access to or utilization of the raw data, obtained ethical approval from the Ethics Committee of Shanghai Pudong Hospital (Shanghai, China, wz.010). The guidelines were outlined, and the procedures were conducted in accordance with the Declaration of Helsinki. All participants were informed and volunteered provided informed consent prior to participating in the study. Informed written consent was obtained from the patients for the publication of this study. All the data used in this study were anonymized before use.

Acknowledgments

Special acknowledgements and sincere gratitude would be extended to all the rotation students who actively participated in this pilot study and who had made significant contributions to their daily job in the Department of Endocrinology at Shanghai Pudong Hospital.

Author Contributions

All authors made significant contributions to this work, including the conception of the study, study design, execution, data acquisition, analysis, and interpretation of data. All participants actively participated in drafting, revising, and critically reviewing the manuscript. The final version of the manuscript was approved by all authors. They collectively agreed to the journal to which the article was submitted and were willing to take full responsibility for all aspects of the work.

Consent for Publication

Not applicable.

Funding

This work was supported by Fudan Good Practice Program of Teaching and Learning (FD2023A227), Fudan Zhangjiang Clinical Medicine Innovation Fund Project (KP0202118), Talents Training Program of Shanghai Pudong Hospital (YQ202101), Project of Key Medical Discipline of Pudong Hospital of Fudan University (Zdxk2020-11), Integrated Traditional Chinese and Western Medicine (YC-2023-0404), Project of Key Medical Specialty and Treatment Center of Pudong Hospital of Fudan University (Zdzk2020-24), Integrative Medicine special fund of Shanghai Municipal Health Planning Committee (ZHYY- ZXYJHZX-2-201712), Special Department Fund of the Pudong New Area Health Planning Commission (PWZzk2017-03), Outstanding Leaders Training Program of Pudong Health Bureau of Shanghai (PWR12014-06), Pudong New Area Clinical Plateau Discipline Project (PWYgy-2021-03), the Natural Science Foundation of China (21675034), National Natural Science Foundation of China (81370932), Shanghai Natural Science Foundation (19ZR1447500), Pudong New Area Clinical Characteristic Discipline Project (PWYts2021-11), Pudong New Area Clinical Characteristic Discipline Project (PWYts2021-01), Wenzhou Medical University Education Grant (JG2021197).

Disclosure

The authors declare that there are no potential conflicts of interest in this work.

References

1. 7. diabetes technology: standards of care in diabetes-2024. Diabetes Care. 2024;47(Suppl 1):S126–s144. doi:10.2337/dc24-S007

2. Kogan JR, Holmboe ES, Hauer KE. Tools for direct observation and assessment of clinical skills of medical trainees: a systematic review. JAMA. 2009;302(12):1316–1326. doi:10.1001/jama.2009.1365

3. Liu CS, Wang YM, Lin HN. An 8-year retrospective survey of assessment in postgraduate dental training in complicated tooth extraction competency. J Dent Sci. 2020;16(3):891–898. doi:10.1016/j.jds.2020.11.006

4. Mallik R, Patel M, Atkinson B, Kar P. Exploring the role of virtual reality to support clinical diabetes training-a pilot study. J Diabetes Sci Technol. 2022;16(4):844–851. doi:10.1177/19322968211027847

5. Nikendei C, Huhn D, Pittius G, et al. Students’ perceptions on an interprofessional ward round training - a qualitative pilot study. GMS J Med Educ. 2016;33(2):Doc14. doi:10.3205/zma001013

6. Sperl-Hillen J, PJ O, Ekstrom HL, et al. Educating resident physicians using virtual case-based simulation improves diabetes management: a randomized controlled trial. Acad med. 2014;89(12):1664–1673. doi:10.1097/ACM.0000000000000406

7. Khan R, Homsi H, Gimpaya N, et al. Validity evidence for observational ERCP competency assessment tools: a systematic review. Endoscopy. 2023;55(9):847–856. doi:10.1055/a-2041-7546

8. Lagoo JY, Joshi SB. Introduction of direct observation of procedural skills (DOPS) as a formative assessment tool during postgraduate training in anaesthesiology: exploration of perceptions. Indian J Anaesth. 2021;65(3):202–209. doi:10.4103/ija.IJA_124_20

9. Siau K, Crossley J, Dunckley P, et al. Colonoscopy direct observation of procedural skills assessment tool for evaluating competency development during training. Am J Gastroenterol. 2020;115(2):234–243. doi:10.14309/ajg.0000000000000426

10. Kogan JR, Hatala R, Hauer KE, Holmboe E. Guidelines: the do’s, don’ts and don’t knows of direct observation of clinical skills in medical education. Perspect Med Educ. 2017;6(5):286–305. doi:10.1007/S40037-017-0376-7

11. Alunno A, Najm A, Sivera F, Haines C, Falzon L, Ramiro S. Assessment of competences in rheumatology training: results of a systematic literature review to inform EULAR points to consider. RMD Open. 2020;6(2):e001330. doi:10.1136/rmdopen-2020-001330

12. El MT, McCrudden R, Shetty D, et al. UK and Ireland Joint Advisory Group (JAG) consensus statements for training and certification in diagnostic endoscopic ultrasound (EUS). Gut. 2023;73(1):118–130. doi:10.1136/gutjnl-2023-329800

13. He Y, Wen S, Zhou M, Li X, Gong M, Zhou L. A pilot study of modified mini-clinical evaluation exercises (Mini-CEX) in rotation students in the department of endocrinology. Diabetes Metab Syndr Obes. 2022;15:2031–2038. doi:10.2147/DMSO.S372253

14. Lugassy D, Herszage J, Pilo R, Brosh T, Censor N. Consolidation of complex motor skill learning: evidence for a delayed offline process. Sleep. 2018;41(9). doi:10.1093/sleep/zsy123

15. Ivers N, Jamtvedt G, Flottorp S, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev. 2012;2012(6):Cd000259. doi:10.1002/14651858.CD000259.pub3

16. Stigt JA, Koele JH, Brand PLP, Jaarsma DAC, Slootweg IA. Workplace mentoring of residents in generic competencies by an independent coach. Perspect Med Educ. 2018;7(5):337–341. doi:10.1007/S40037-018-0452-7

17. Weller J, Frengley R, Torrie J, et al. Evaluation of an instrument to measure teamwork in multidisciplinary critical care teams. BMJ Qual Saf. 2011;20(3):216–222. doi:10.1136/bmjqs.2010.041913

18. Iyasere CA, Wing J, Martel JN, Healy MG, Park YS, Finn KM. Effect of increased interprofessional familiarity on team performance, communication, and psychological safety on inpatient medical teams: a randomized clinical trial. JAMA Intern Med. 2022;182(11):1190–1198. doi:10.1001/jamainternmed.2022.4373

19. Salmon P, Young B. Creativity in clinical communication: from communication skills to skilled communication. Med Edu. 2011;45(3):217–226. doi:10.1111/j.1365-2923.2010.03801.x

20. Verheijden M, Giroldi E, van den Eertwegh V, et al. Identifying characteristics of a skilled communicator in the clinical encounter. Med Edu. 2023;57(5):418–429. doi:10.1111/medu.14953

21. Forsey J, Ng S, Rowland P, Freeman R, Li C, Woods NN. The basic science of patient-physician communication: a critical scoping review. Acad med. 2021;96(11s):S109–s118. doi:10.1097/ACM.0000000000004323

22. Li M, Kurahashi AM, Kawaguchi S, Siemens I, Sirianni G, Myers J. When words are your scalpel, what and how information is exchanged may be differently salient to assessors. Med Edu. 2024;58(11):1324–1332. doi:10.1111/medu.15458

23. Alahdab F, Halvorsen AJ, Mandrekar JN, et al. How do we assess resilience and grit among internal medicine residents at the mayo clinic? A longitudinal validity study including correlations with medical knowledge, professionalism and clinical performance. BMJ open. 2020;10(12):e040699. doi:10.1136/bmjopen-2020-040699