Several machine learning algorithms have demonstrated high predictive capability in the identification of cancer within digitized pathology slides. The Augmented Reality Microscope (ARM) has allowed these algorithms to be seamlessly integrated within the current pathology workflow by overlaying their inferences onto its microscopic field of view in real time. In this paper, we present an independent assessment of the LYmph Node Assistant (LYNA) models, state-of-the-art algorithms for the identification of breast cancer metastases in lymph node biopsies which have been optimized for usage at different ARM magnifications. We assessed the models on a set of 40 whole slide images at the commonly used objective magnifications of 10x, 20x, and 40x. We analyzed the performance of the models across clinically relevant subclasses of tissue, including metastatic breast cancer, lymphocytes, histiocytes, veins, and fat. We also analyzed the models' performance on potential types of contaminant tissue such as endometrial carcinoma and papillary thyroid cancer. Each model obtained overall AUC values of approximately 0.98, accuracy values of approximately 0.94, and sensitivity values above 0.88 at classifying small regions of a field of view as benign or cancerous. Across tissue subclasses, the models performed most accurately on fat and blood, and least accurately on histiocytes, germinal centers, and sinus. The models also struggled with the identification of isolated tumor cells, especially at lower magnifications. After testing, we manually reviewed the discrepancies between model predictions and ground truth in order to understand the causes of error. We introduce a distinction between proper and improper ground truth to allow for analysis in cases of uncertain annotations or on tasks with low inter-rater reliability. Taken together, these methods comprise a novel approach for exploratory model analysis over complex anatomic pathology data in which precise ground truth is difficult to establish.

The authors have declared no competing interest.

The Defense Innovation Unit provided support for this study under contract #W56KGU-21-F- 0008. The Joint Artificial Intelligence Center provided support for this study under contract #W56KGU-18-D-0004. Authors from The Henry M. Jackson Foundation for the Advancement of Military Medicine were supported by NCRADA-16-471 and NCRADA-NMCSD-21-536.

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

Yes

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

IRB of Naval Medical Center San Diego gave ethical approval for this work. Protocol number NHG.2018.0001.

I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.

Yes

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

Yes

I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.

Yes

The datasets used and analyzed in this study are available from the corresponding author upon reasonable request.