The state-of-the-art in automatically segmenting organs from abdominal CT images are supervised deep learning approaches (Gibson et al., 2018, Kim et al., 2021, Ma et al., 2022). However, training a segmentation model for all abdominal organs and tumors is challenging, as there are no public data sets containing a large number of CT scans accompanied with complete segmentation labels. Existing datasets are only partially labeled, i.e., only a few organs or types of tumors are segmented (Bilic et al., 2023, Simpson et al., 2019). Accordingly, models specifically suitable for being trained on partially labeled CT data from multiple institutes have been proposed (Zhang et al., 2021b, Dmitriev and Kaufman, 2019).

A straightforward approach for segmenting multiple organs in CT is to use segmentation models that are separately trained on each partially labeled CT dataset (Hu et al., 2016). However, this strategy is computationally inefficient and accuracy is limited when the dataset includes a small number of samples. An alternative is incremental learning (Li and Hoiem, 2017, Elskhawy et al., 2020, Vu et al., 2021), where a single model is updated by training it sequentially on the datasets. Despite recent advances, these models often struggle to maintain accuracy on the initial data sets, since they easily forget the knowledge gained from the previously used data sets (Xiao et al., 2023). One promising training strategy is federated learning (Li et al., 2020a), which trains a model at each site and then merges the parameters of the model across sites via a central server.

Though federated learning has been applied to medical image segmentation, most existing implementations assume that all sites (a.k.a. client nodes) have labels for the same set of organs (Li et al., 2019, Wang et al., 2020, Xia et al., 2021). In practice, however, the CT acquisitions and the organs annotated differ across sites (or nodes). To account for this difference, one can train a network of nodes (Xu et al., 2023) using federated averaging (McMahan et al., 2017) (FedAvg), i.e., each node has its own encoder and a shared decoder is used across all nodes. However, this approach requires a large number of parameters to be tuned and the accuracy is limited since each encoder is trained on a relatively small number of samples. In addition, the knowledge for segmenting organs is lost during the ‘local’ training at a node (a.k.a. catastrophic forgetting), which is specific to certain organs. Finally, the isolated updating of local models can result in degraded accuracy of the overall model across sites.

To overcome this limitation, we regularize training at each site with knowledge from the global model and pre-trained organ-specific segmentation models. We do so using global and local knowledge distillation (KD) (Gou et al., 2021), that effectively mitigate forgetting by imposing constraints to retain segmentation results for unlabeled organs when trained with partially labeled data. Furthermore, we propose a new baseline structure that consists of a shared encoder–decoder, similar to U-Net (Ronneberger et al., 2015), and lightweight segmentation heads with just 162 parameters for each target organ. As our model shares most parts of the encoder and decoder across sites, the representations from multiple datasets can be accomplished in a single model and the inference can be quickly performed without repeating the feed-forward process for multiple target organs (Zhang et al., 2021a, Dmitriev and Kaufman, 2019, Wu et al., 2022). We evaluate our proposed method using eight public CT datasets (Bilic et al., 2023, Heller et al., 2019, Simpson et al., 2019, Landman et al., 2015). The datasets differ with respect to the segmented organs and pathology. Besides achieving significantly higher accuracy than several state-of-the-art methods, our method is efficient with respect to inference time, contains a relatively small number of parameters, and is robust against catastrophic forgetting.