Birkmeyer JD, Finks JF, O’Reilly A, Oerline M, Carlin AM, Nunn AR, Dimick J, Banerjee M, Birkmeyer MJO (2013) Surgical skill and complication rates after bariatric surgery. N Engl J Med 369:1434–1442. https://doi.org/10.1056/nejmsa1300625

Article  CAS  PubMed  Google Scholar 

Schlomm T, Huland H, Graefen M (2014) Improving outcome of surgical procedures is not possible without adequate quality measurement. Eur Urol. https://doi.org/10.1016/j.eururo.2013.11.042

Article  PubMed  Google Scholar 

Tam V, Zeh HJ, Hogg ME (2017) Incorporating metrics of surgical proficiency into credentialing and privileging pathways. JAMA Surg. https://doi.org/10.1001/jamasurg.2017.0025

Article  PubMed  Google Scholar 

Reznick RK (1993) Teaching and testing technical skills. Am J Surg 165(3):358–361. https://doi.org/10.1016/S0002-9610(05)80843-8

Article  CAS  PubMed  Google Scholar 

Goh AC, Goldfarb DW, Sander JC, Miles BJ, Dunkin BJ (2012) Global evaluative assessment of robotic skills: validation of a clinical assessment tool to measure robotic surgical skills. J Urol 187:247–252. https://doi.org/10.1016/j.juro.2011.09.032

Article  PubMed  Google Scholar 

Martin JA, Regehr G, Reznick R, Macrae H, Murnaghan J, Hutchison C, Brown M (1997) Objective structured assessment of technical skill (OSATS) for surgical residents. Br J Surg 84:273–278. https://doi.org/10.1002/bjs.1800840237

Article  CAS  PubMed  Google Scholar 

Hussein AA, Ghani KR, Peabody J, Sarle R, Abaza R, Eun D, Hu J, Fumo M, Lane B, Montgomery JS, Hinata N, Rooney D, Comstock B, Chan HK, Mane SS, Mohler JL, Wilding G, Miller D, Guru KA (2017) Development and validation of an objective scoring tool for robot-assisted radical prostatectomy: prostatectomy assessment and competency evaluation. J Urol 197:1237–1244. https://doi.org/10.1016/j.juro.2016.11.100

Article  PubMed  Google Scholar 

Raza SJ, Field E, Jay C, Eun D, Fumo M, Hu JC, Lee D, Mehboob Z, Nyquist J, Peabody JO, Sarle R, Stricker H, Yang Z, Wilding G, Mohler JL, Guru KA (2015) Surgical competency for urethrovesical anastomosis during robot-assisted radical prostatectomy: development and validation of the robotic anastomosis competency evaluation. Urology 85(1):27–32. https://doi.org/10.1016/j.urology.2014.09.017

Article  PubMed  Google Scholar 

Wang C, Han L, Stein G, Day S, Bien-Gund C, Mathews A, Ong JJ, Zhao PZ, Wei SF, Walker J, Chou R, Lee A, Chen A, Bayus B, Tucker JD (2020) Crowdsourcing in health and medical research: a systematic review. Infect Dis Poverty 9:1–9. https://doi.org/10.1186/s40249-020-0622-9

Article  Google Scholar 

White LW, Kowalewski TM, Dockter RL, Comstock B, Hannaford B, Lendvay TS (2015) Crowd-sourced assessment of technical skill: a valid method for discriminating basic robotic surgery skills. J Endourol 29(11):1295–1301. https://doi.org/10.1089/end.2015.0191

Article  PubMed  Google Scholar 

Vernez SL, Huynh V, Osann K, Okhunov Z, Landman J, Clayman RV (2017) C-SATS: assessing surgical skills among urology residency applicants. J Endourol 31:S95–S100. https://doi.org/10.1089/end.2016.0569

Article  PubMed  Google Scholar 

Dai JC, Lendvay TS, Sorensen MD (2017) Crowdsourcing in surgical skills acquisition: a developing technology in surgical education. J Grad Med Educ 9:697–705. https://doi.org/10.4300/JGME-D-17-00322.1

Article  PubMed  PubMed Central  Google Scholar 

Polin MR, Siddiqui NY, Comstock BA, Hesham H, Brown C, Lendvay TS, Martino MA (2016) Crowdsourcing: a valid alternative to expert evaluation of robotic surgery skills. Am J Obstet Gynecol 215:644.e1-644.e7. https://doi.org/10.1016/j.ajog.2016.06.033

Article  PubMed  Google Scholar 

Rhienmora P, Haddawy P, Suebnukarn S, Dailey MN (2011) Intelligent dental training simulator with objective skill assessment and feedback. Artif Intell Med 52(2):115–121. https://doi.org/10.1016/j.artmed.2011.04.003

Article  PubMed  Google Scholar 

Jarc AM, Curet MJ (2017) Viewpoint matters: objective performance metrics for surgeon endoscope control during robot-assisted surgery. Surg Endosc 31:1192–1202. https://doi.org/10.1007/s00464-016-5090-8

Article  PubMed  Google Scholar 

Jun SK, Narayanan MS, Agarwal P, Eddib, A, Singhal, P, Garimella, S, Krovi, V (2012) Robotic minimally invasive surgical skill assessment based on automated video-analysis motion studies. In: Proceedings of the IEEE RAS and EMBS International Conference on Biomedical Robotics and Biomechatronics, 25–31 https://doi.org/10.1109/BioRob.2012.6290869

Brown KC, Bhattacharyya KD, Kulason S, Zia A, Jarc A (2020) How to bring surgery to the next level: Interpretable skills assessment in robotic-assisted surgery. Visc Med 36:463–470. https://doi.org/10.1159/000512437

Article  PubMed  PubMed Central  Google Scholar 

Hung AJ, Chen J, Jarc A, Hatcher D, Djaladat H, Gill IS (2018) Development and validation of objective performance metrics for robot-assisted radical prostatectomy: a pilot study. J Urol 199:296–304. https://doi.org/10.1016/j.juro.2017.07.081

Article  PubMed  Google Scholar 

Chen AB, Liang S, Nguyen JH, Liu Y, Hung AJ (2021) Machine learning analyses of automated performance metrics during granular sub-stitch phases predict surgeon experience. Surgery 169(5):1245–1249. https://doi.org/10.1016/j.surg.2020.09.020

Article  PubMed  Google Scholar 

Cacciamani GE, Anvar A, Chen A, Gill I, Hung AJ (2021) How the use of the artificial intelligence could improve surgical skills in urology: state of the art and future perspectives. Curr Opin Urol 31(4):378–384

Article  PubMed  Google Scholar 

Hung AJ, Oh PJ, Chen J, Ghodoussipour S, Lane C, Jarc A, Gill I (2019) Experts vs super-experts: differences in automated performance metrics and clinical outcomes for robot-assisted radical prostatectomy. BJU Int 123:861–868. https://doi.org/10.1111/bju.14599

Article  PubMed  Google Scholar 

Hung A, Chen J, Che Z, Nilanon T, Jarc A, Guo L, Oh P, Gill I, Liu Y (2018) PD58-01 utilization of machine learning and automated performance metrics to evaluate robot-assisted radical prostatectomy performance and predict patient outcomes. J Urol 199(4):e1133. https://doi.org/10.1016/j.juro.2018.02.2789

Article  Google Scholar 

Hung AJ, Ma R, Cen S, Nguyen JH, Lei X, Wagner C (2021) Surgeon automated performance metrics as predictors of early urinary continence recovery after robotic radical prostatectomy—a prospective bi-institutional study. Eur Urol Open Sci 27:65–72. https://doi.org/10.1016/j.euros.2021.03.005

Article  PubMed  PubMed Central  Google Scholar 

Ghodoussipour S, Reddy SS, Ma R, Ma R, Huang D, Nguyen J, Hung AJ (2021) An objective assessment of performance during robotic partial nephrectomy: validation and correlation of automated performance metrics with intraoperative outcomes. J Urol 205(5):1294–1302. https://doi.org/10.1097/JU.0000000000001557

Article  PubMed  Google Scholar 

Trinh L, Mingo S, Vanstrum EB, Sanford DI, Ma R, Nguyen JH, Liu Y, Hung AJ (2022) Survival Analysis using surgeon skill metrics and patient factors to predict urinary continence recovery after robot-assisted radical prostatectomy. Eur Urol Focus 8(2):623–630. https://doi.org/10.1016/j.euf.2021.04.001

Article  PubMed  Google Scholar 

Tang B, Hanna GB, Joice P, Cuschieri A (2004) Identification and categorization of technical errors by observational clinical human reliability assessment (OCHRA) during laparoscopic cholecystectomy. Arch Surg 139(11):1215–1220

Article  CAS  PubMed  Google Scholar 

Ahmed K, Miskovic D, Darzi A, Athanasiou T, Hanna GB (2011) Observational tools for assessment of procedural skills: a systematic review. Am J Surg 202:469-480.e6

Article  PubMed  Google Scholar 

Biggs J (1996) Enhancing teaching through constructive alignment. High Educ 32(3):347–364

Article  Google Scholar 

Nazari T, Bogomolova K, Ridderbos M, Dankbaar MEW, van Merriënboer JJG, Lange JF, Wiggers T, Wiggers JA (2021) Global versus task-specific postoperative feedback in surgical procedure learning. Surgery 170:81–87. https://doi.org/10.1016/j.surg.2020.12.038

Article  PubMed  Google Scholar 

Lavanchy JL, Zindel J, Kirtac K, Twick I, Hosgor E, Candinas D, Beldi G (2021) Automation of surgical skill assessment using a three-stage machine learning algorithm. Sci Rep 11:1–9. https://doi.org/10.1038/s41598-021-84295-6

Article  CAS  Google Scholar 

Zia A, Hung A, Essa I, Jarc A (2018) Surgical activity recognition in robot-assisted radical prostatectomy using deep learning. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Hashimoto DA, Rosman G, Witkowski ER, Stafford C, Navarette-Welton AJ, Rattner DW, Lillemoe KD, Rus DL, Meireles OR (2019) Computer vision analysis of intraoperative video: automated recognition of operative steps in laparoscopic sleeve gastrectomy. Ann Surg 270(3):414–421.