Surgery represents a vital part of the healthcare system, with over 300 million surgical procedures performed worldwide annually [1]. Improvements in perioperative care protocols, such as the enhanced recovery after surgery (ERAS) protocol, as well as the adoption of minimally invasive surgical (MIS) techniques like robotic surgery, have improved surgery outcomes [2], [3], [4], [5], [6], [7]. However, surgery still maintains significant morbidity and leads to a significant proportion (over 40%) of preventable medical errors, suggesting the need for improvements in surgical safety and training [8].

Analysis and education of surgical procedures through videos has become an important strategy for improving surgical care. This is especially accessible for MIS, which necessitates visualization with fiber optic cameras making for straightforward recording and review [9]. Literature on video-based study and assessment has shown that differences in intraoperative surgical proficiency impact patient outcomes [10]. Video-based assessment (VBA) from surgical videos has important implications for surgical assessment, feedback, training, and competency evaluation. Multiple direct surgical video observation-based robotic surgery competency metrics exist, like the Global Assessment of Robotic Skills (GEARS). However, all necessitate time-intensive evaluations by surgeons and are, at their core, subjective with inter-rater variability [11,12].

Artificial intelligence (AI) has recently become more relevant in the medical field. Computer vision (CV) is a field of AI that enables computer systems to derive meaningful information from digital images, videos, or other visual inputs. CV uses deep learning and convolutional neural networks to discern the qualities of visual media [13]. This technology can potentially revolutionize how surgical video is analyzed and utilized in the future. We can envision a future where surgeons and trainees can receive an automatic assessment of surgical performance with specific feedback from a CV system, supporting intraoperative training and decision-making. CV has been used in fields such as gastroenterology with promising results, with one study successfully designing visual markers that could be detected with CV during colonoscopy to help avoid missing colonic polyps, and another distinguishing diminutive from hyperplastic polyps [14], [15], [16]. While most of the focus of CV in surgery has been on retrospective image and video analysis [17], AI is now being used to provide real-time feedback to the operator. This real-time monitoring has the potential to aid in surgical education, quality improvement initiatives, improve outcomes, and avoid adverse events [17]. Annotation is an important prerequisite to real-time surgical monitoring, as technology must show an ability to label operative events accurately to provide any clinically significant outputs. Studies note many challenges with surgical annotation, including the accuracy of spatial and temporal information and discerning clinically meaningful events [13]. However, recent advances have been made. One study used CV to develop a framework for semiautomated expert annotation in gastroenterology [18]. Multiple studies in general surgical procedures such as cholecystectomy and laparoscopic sleeve gastrectomy have used retrospective image and video analysis to develop machine learning networks for identification of surgical phases of these operations [17,19]. Our objective was to explore the potential of computer vision and artificial intelligence in real-time surgical video analysis during urologic robotic surgery. We successfully describe the first-ever use of real-time AI-supported surgical annotations establishing their potential role in education and providing intraoperative decision support.

To capture live surgical videos and successfully annotate key surgical steps and safety milestones in real-time intraoperative decision support, we demonstrate the first use of CV in urologic robotic surgery to capture live surgical videos and successfully annotate key surgical steps and safety milestones in real-time.