The management of a variety of vision-threatening retinal conditions can be substantially improved with the aid of imaging technologies that reveal lesions as diagnostic and prognostic imaging biomarkers. Optical Coherence Tomography (OCT) is one such imaging modality, providing high-resolution, cross-sectional images of the retina for improved detection and monitoring of retinal diseases. In clinical practice, the ability to recognize these lesions can facilitate treatment planning. Moreover, segmentation and quantification of these imaging biomarkers have been shown to provide a further nuanced understanding of their contribution to disease activity (Schmidt-Erfurth et al., 2021).

Semantic segmentation is one of the fundamental computer vision tasks that aim to obtain pixel-level segmentation results for given images. However, annotating medical images at a pixel-level can be time-consuming and costly, especially for biomedical images that typically require domain knowledge. To overcome this barrier, efforts have been dedicated to weakly supervised semantic segmentation (WSSS). WSSS uses weaker forms of supervision, such as image-level labels (Pinheiro and Collobert, 2015, Ahn and Kwak, 2018, Kolesnikov and Lampert, 2016, Kwak et al., 2017, Niu et al., 2023), scribbles (Lin et al., 2016, Vernaza and Chandraker, 2017, Luo et al., 2022, Valvano et al., 2021), and bounding boxes (Kervadec et al., 2019, Ma et al., 2022, Oh et al., 2021, Dai et al., 2015), to create pixel-level predicted segmentation. One common approach of WSSS is to create pseudo labels for training segmentation networks using classification task byproducts, such as Class Activation Maps (CAMs) (Zhou et al., 2016), which provide a heatmap of salient regions for the predicted target class. In this study, we focus on developing a WSSS method that only utilizes image-level supervision to segment lesions in multi-label OCT images.

In recent years, various CAM-based WSSS methods have been proposed (Wu et al., 2021, Ahn and Kwak, 2018, Wu et al., 2021, Chen et al., 2022b). However, the focus has mainly been on natural images, and applying these models directly to medical datasets can be problematic due to the inherent differences between natural and medical images. These differences include variations in image intensities, object appearance, and diverse scales of anatomical structures (Xing et al., 2021, van Engeland et al., 2006, Prince and Links, 2006, Chen et al., 2022a). Particularly, the extracted raw and refined CAMs, which are generated by extending seed CAMs to entire objects, are still relatively coarse due to the small, noisy, and low-contrast lesions in OCT images compared to natural objects.

In medical imaging, especially in OCT images, current CAM-based methods (Wang et al., 2021, Roth et al., 2021, Zhang et al., 2022, Liu et al., 2023) face two primary challenges: they are often tuned for specific diseases or simplified to binary segmentation per image, and struggle to detect all lesion regions when those regions are low-contrast. Recognizing the importance of detecting subtle or unexpected abnormalities, which can indicate underlying pathologies, anomaly detection has become a popular research direction to highlight abnormal regions (Schlegl et al., 2019, Zhou et al., 2020, Liu et al., 2023). However, the limitation of anomaly detection is its inability to differentiate between lesion types.

Drawing inspiration from anomaly detection and CAM-based WSSS, our work seeks to integrate abnormal signals with CAMs in multi-class WSSS, which remains underexplored in the literature to the best of our knowledge. We propose an anomaly-guided mechanism (AGM) in this paper that can capture rich anomalous information from a multi-label OCT image. In particular, our proposed method exploits the anomaly-discriminative representation with the aid of GAN-generated healthy counterpart of the same retina (Akcay et al., 2018) to provide a more robust representation of the lesion. We further enhance the model’s ability to localize small lesions with spatial constraints by incorporating self-attention, and an iterative refinement learning step to leverage anomalous features. Fig. 1 illustrates the effectiveness of AGM on small low-contrast lesions in comparison with a plain backbone without anomaly guidance. In summary, our main contributions are threefold:

We introduce a novel anomaly-guided WSSS method with image-level supervision specifically designed for medical lesion segmentation.

We leverage anomaly information for the detection of small lesions. Instead of using anomaly knowledge in the pre/post-processing, we utilize the self-attention mechanism to enhance small lesion localization by capturing global lesion information and develop an efficient refinement learning approach to further direct the attention by utilizing anomalous features.

We perform comprehensive experiments on two public and one private OCT datasets, achieving superior performance compared to current state-of-the-art methods in lesion segmentation with image-level labels only.