Patients

This study was approved by the Institutional Review Board, Taipei Veterans General Hospital, which complied with standards of the Declaration of Helsinki and current ethical guidelines. Due to the retrospective nature of the study, the Institutional Review Board waived the need for written informed consent. The identifying information of the enrolled subjects has been delinked and therefore authors could not access the information.

From October 1, 2007 to August 31, 2019, 884 consecutive HCC patients receiving surgical resection and radiofrequency ablation (RFA) in Taipei Veterans General Hospital with available CT images before treatment were retrospectively screened. The inclusion criteria were: (1) Age ≥ 20 years; (2) Available CT image within 3 months prior to surgical resection or RFA; (3) No other loco-regional treatment prior to surgical resection or RFA. Patients were excluded by the following criteria: (1) without complete dynamic contrast-enhanced CT (CECT) images including non-contrast phase, arterial phase and portal venous phase (n = 87); (2) poor image quality or unable to align the dynamic CT images according to the Z-axis (n = 202). Finally, 595 HCC patients with complete dynamic CECT images were enrolled in this study. HCC was diagnosed before surgery or RFA by CECT or magnetic resonance imaging (MRI), which fulfilled the diagnostic criteria of the American Association for the Study of Liver Diseases (AASLD) treatment guidelines for HCC [14] or was confirmed pathologically after surgery and RFA.

CECT image segmentation

The image acquisition protocols of CT scanners involved in the present study are shown in Table S1. Interpretation and tumor segmentation of all CECT images of the 595 patients were performed by three radiologists who were blinded to the clinical and pathological data. The labeling of the HCC in dynamic CT images included the non-contrast phase, arterial phase and portal-venous phase. The three radiologists had read > 2,000 liver CT studies per year for at least 5 years. When contouring the tumor, the edge of the observed focal lesion within the liver was defined as an imaging appearance that is distinctive from the background according to the Liver Reporting and Data System (LI-RADS) [2, 15]. The contours of the liver were also labelled in 200 cases for training of segmentation of the liver. For evaluation of the HFS-Net model, ground truth tumor compartments were delineated manually. This was performed using a semiautomatic approach with subsequent manual editing (IntelliSpace Discovery; Philips Healthcare, Netherlands), performed by the three experienced radiologists [5].

CT image processing

The characteristics of the CT image dataset were shown in Table 1. The original CT images were all 3D images stacked by 512 × 512 2D slices. Before the experiment, this study used downsampling to reduce all slices into a size of 256 × 256 and adjusted all CT image scale to 1 pixel equal to 1.4 mm. We aligned the dynamic CT images according to the Z-axis coordinates of the slices and superimposed them into a three-channel image for use. After alignment, 24,810 images (a total of 74,430 slices) were enrolled for analysis.

Table 1 Characteristics of the hepatocellular carcinomaStudy design for HFS-Net

To design the best architecture of HFS-Net for HCC detection and segmentation, an ablation study was first conducted for evaluating various architectures of deep learning models. We randomly selected 491 cases from a total of 595 patients with a broad distribution of various tumor sizes was conducted for the ablation study.

The whole dataset was used to design and evaluate HFS-Net, which was randomly divided into non-overlapping training, validation, and test datasets in a 5:2:3 ratio. The training, validation, and test sets have 298, 118, and 179 patients, respectively. Each patient had a different number of slices and all were analyzed. The training set is used for constructing the HFS-Net model, while the validation set is used for tuning and optimization of model parameters. We then evaluate the performance of the HFS-Net model on a separate test set. HFS-Net was expected to combine the best sub-models by thorough evaluation of the candidate sub-models.

Therefore, the splitting strategy of the data and the construction of the model are independent in the two stages, and the purposes in the two stages are different. There is no comparison between the experimental results from the two stages, and the constructed models in the ablation experiments are not used in the final HFS-Net model. Instead, it uses the conclusions drawn from the ablation experiments as the background knowledge for model construction.

Ablation study design for HFS-Net development

The ablation study investigates the performance of the proposed AI system HFS-Net by evaluating certain sub-models to understand the contribution and tolerance of the sub-models to the overall system. This study aims to discern the individual and combined strength of various sub-models in various scenarios. The influencing factors of system performance include architectures (U-Net, DenseU-net, Hyper-DenseU-Net, and R2U-Net), phases (non-contrast, arterial, portal-venous, and dynamic phases), loss functions (cross entropy, focal loss, median frequency balance, and dice loss), tumor sizes (the longest tumor axes in slices), assessment (dice score for model’s segmentation ability and tumor detection rate) and the fusion strategy.

The final stage of the HFS-Net design involves a comparison between 2D HFS-Net and the more comprehensive 3D HFS-Net. 2D HFS-Net leveraged the initial stages of hierarchical architecture and 2-D spatial features, while 3D HFS-Net incorporates a complete hierarchical architecture with fusion strategy and 3-D spatial features. This comparison helps to thoroughly evaluate the impact of hierarchical structures and fusion strategies to enhance the segmentation and detection of HCC.

Neural network architecture of HFS-Net

Figure 1 showed the data flow of our proposed HFS-Net method for liver and tumor detection and segmentation according to results of the ablation study. We cascaded five sub-models trained by different learning strategies (Table 2). HFS-Net consisted of three stages based on U-Net and DenseU-Net. The first stage of HFS-Net is the liver and tumor segmentation with tumor size estimation using 2D DenseU-Net, where every slice of the entire CT scan case was taken as input. The second stage is the divide-and-conquer stage. According to the tumor size calculated in the first stage, the tumors with the longest axis less than or equal to m pixels in the slice are assigned to the smaller tumor group, and those with more than m pixels are assigned to the larger tumor group. This study uses m = 30 pixels as the demarcation point, which is about 4.2 cm. In the divide-and-conquer stage, we use a customized model to adaptively segment large and small tumors in slices. For small-tumor groups, 2D DenseU-Net uses dynamic CT images to segment tumors, and for large-tumor groups, 2D U-Net uses portal-venous phase CT images to segment tumors. The third stage is the fusion strategy stage. This stage integrates the portal-venous phase CT image, the outcomes of the previous two stages and the segmentation of ​​the liver, and uses 3D U-Net to segment the final result of the 3D liver tumor. The detailed modeling and learning strategies of HFS-Net are described in Supplementary Methods.

Fig. 1

The HFS-Net data flow for liver and tumor segmentation. Stage I: Identify tumor’s longest axis in every slice of a case. Stage II: Accord tumor size by feeding the CT slice which the longest axis of tumors over 30 pixels in the slice for flarge computation and which the longest axis less than 30 pixels of tumors in the slice for fsmall computation. Stage III: Combine venous phases of CT images and results of fliver, fsize, flarge, and fsmall as input of f3D computation for getting final segmentation of tumors

Table 2 Function strategy in the HFS-NetEvaluation of the performance of the HFS-Net model

We used dice per case and dice global of Jaccard similarity as metrics to evaluate the performance of segmentation. Dice per case represents the average dice of the case, and dice global is the dice score that combines all the slices as a case to calculate. We used sensitivity, precision, and F1-score to evaluate detection performance. When evaluating the performance of detection, the criterion for successful detection is defined as the dice global that overlaps the model results with the corresponding tumor labels exceeding θ. Otherwise, it is a false positive. If the model fails to detect a tumor in a slice, it is considered a false negative. We set θ = 0.2 to require a significant, but not exact overlap as post study did [16]. In order to analyze the detection effect of the model more deeply, we designed four indicators to evaluate the detection performance of the model, including overall detection performance of tumors in slices (Per tumor volume), mean detection performance of tumors in slices (Per tumor cut), abnormal slice detection (Per slice) and abnormal case detection (Per patient).

The overall detection performance of tumors in slices means to use the tumors in all slices as the denominator to evaluate the performance of tumor detection. The mean detection performance of tumors in slices means to use the tumors in all slices of the case as the denominator to evaluate the performance of each case on average. Abnormal slice detection means the ability to detect at least one of the tumors in a slice with tumors, and abnormal case detection means the ability to detect at least one of the tumors in a case with tumors.

Experiment environment

In this study, we used Nvidia GeForce RTX 2080 Ti (12GB) as the GPU, Intel(R) Xeon(R) Gold 6136 CPU @ 3.00 Ghz as the CPU, and used the CentOS Linux release 8.3.2011 as operating system. The method was implemented with python3.7 and Pytorch packages [17].