The American College of Surgeons (ACS) / Association of Program Directors in Surgery (APDS) Resident Skills Curriculum is composed of three phases. Phase one includes technical skills including handsewn bowel anastomosis and vascular anastomosis. More than 75% of general surgery residencies within the United States complete this curriculum.1, 2, 3 The curriculum includes an Objective Structured Assessment of Technical Skills (OSATS) for which both task-specific checklists and global rating scales (GRS) are completed by raters. These assessments provide residents with the opportunity to receive formative feedback on their technical performance. However, prior work has shown that the technical skill ratings are not necessarily reliable, and faculty graders may benefit from rater training.4

To achieve reliable assessment data on residents’ operative skill performances, it has been shown that faculty should complete 2.3 ratings per resident, or seven faculty should be used for consensus setting via a modified Delphi process.5,6 For more specific technical skill evaluations, prior recommendations outline a minimum of 5to 12 independent faculty judges for global ratings in order to reliably determine technical skill proficiency.7

Establishing an efficient process that minimizes threats to technical skill assessment validity will become particularly important as board certification processes consider adding a technical skill requirement. While the ABS requires that general surgery residents receive six operative and clinical performance evaluations prior to graduation, systems for regular monitoring of the performance ratings to ensure their reliability and validity have not yet been implemented; rather, local programs are responsible for ensuring the accuracy of the feedback.8 In 2014, the American Board of Colon and Rectal Surgery (ABCRS) included a 8-task Colorectal OSATS as a mandatory component of the ABSCRS certification process. Through this assessment process, ABCRS found a poor correlation between trainees’ performance on the OSATS and their oral examination scores, suggesting an unmet need for valid and reliable technical skill assessments prior to final surgical board certification.9 Other surgical fields, including vascular surgery and minimally invasive surgery are also considering adding a technical skill assessment to their board certification processes, however the standard setting and validity tests have not yet been delineated. Vascular surgery is considering the rollout of Fundamentals of Vascular Surgery (FVS) and the Society of American Gastrointestinal and Endoscopic Surgery (SAGES) is considering implementing a requirement for all graduating minimally invasive surgery fellows to submit a video-recorded performance which will be rated by experts as part of the board certification process.10

As the surgical education community begins planning for the addition of technical skill evaluations to high-stakes assessment pathways, the ACS/APDS Resident Skills Curriculum OSATS offer an opportunity to both assess trainees’ technical skill proficiency and to establish a process for standard setting that extends beyond prior work on operative-based assessments. To address the need for rater training, we set out to: (1) develop rater training modules for ACS/ APDS Resident Skills Curriculum technical skills, (2) establish consensus standard scores for video-recorded technical skill performances, and (3) assess the validity impact of the rater training intervention.