Scheme 1. Flowchart of the method used in this study to evaluate robustness of radiomic features. Three 3D ROIs have been delineated on each subject included in the study. Radiomic features from the original image and from the wavelet transformed have been computed using different values of four parameters: bin-width, pixel-distance, interpolator and isotropic resolution. ICC has been used for robustness evaluation and variability of ICC across a parameter was used as an index of ‘sensitivity’ of the feature to the parameter.

Scheme 1. Flowchart of the method used in this study to evaluate robustness of radiomic features. Three 3D ROIs have been delineated on each subject included in the study. Radiomic features from the original image and from the wavelet transformed have been computed using different values of four parameters: bin-width, pixel-distance, interpolator and isotropic resolution. ICC has been used for robustness evaluation and variability of ICC across a parameter was used as an index of ‘sensitivity’ of the feature to the parameter.

Figure 1. Illustrative examples of manual segmentations. For each patient on a row, different segmentations are showed superimposed on CT data: (a) polygonal, (b) free-hand accurate and (c) free-hand rough.

Figure 1. Illustrative examples of manual segmentations. For each patient on a row, different segmentations are showed superimposed on CT data: (a) polygonal, (b) free-hand accurate and (c) free-hand rough.

Figure 2. Robustness of shape features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides 14 shape features. The robustness is above 0.60 for all of them. However, three of them are less robust (<0.90) across segmentations (Sphericity, SurfaceVolumeRatio and Elongation); this might be because they largely depend on the precise ROI contours. They do not depend on bin-width, distance, resolution or interpolator. With the exception of Flatness, the other features have a robustness greater than 0.95.

Figure 2. Robustness of shape features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides 14 shape features. The robustness is above 0.60 for all of them. However, three of them are less robust (<0.90) across segmentations (Sphericity, SurfaceVolumeRatio and Elongation); this might be because they largely depend on the precise ROI contours. They do not depend on bin-width, distance, resolution or interpolator. With the exception of Flatness, the other features have a robustness greater than 0.95.

Figure 3. Robustness of first-order histogram features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides 18 first-order features. The robustness (ICC) is above 0.8 only for skewness, kurtosis, uniformity and 90 percentile. The latter has ICC > 0.90. They are not dependent on bin-width (only entropy has slight dependence). Because they are first-order, they are not dependent on distance, neither do they depend on resolution or interpolator.

Figure 3. Robustness of first-order histogram features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides 18 first-order features. The robustness (ICC) is above 0.8 only for skewness, kurtosis, uniformity and 90 percentile. The latter has ICC > 0.90. They are not dependent on bin-width (only entropy has slight dependence). Because they are first-order, they are not dependent on distance, neither do they depend on resolution or interpolator.

Figure 4. Robustness of second-order GLCM features across segmentations. Pyradiomics computes 24 second-order GLCM features. The robustness (ICC) is above 0.8 per all values of distance, only for Autocorrelation, Idn, Idmn, SumAverage, JointAverage, InverseVariance, JointEnergy, MaximumProbability, Id and Idm. Robustness of many features across segmentations is dependent on bin-width and distance (the vertical dashed red line on the right indicates the 0.9 threshold).

Figure 4. Robustness of second-order GLCM features across segmentations. Pyradiomics computes 24 second-order GLCM features. The robustness (ICC) is above 0.8 per all values of distance, only for Autocorrelation, Idn, Idmn, SumAverage, JointAverage, InverseVariance, JointEnergy, MaximumProbability, Id and Idm. Robustness of many features across segmentations is dependent on bin-width and distance (the vertical dashed red line on the right indicates the 0.9 threshold).

Figure 5. Robustness of second-order GLDM features across segmentations. Pyradiomics computes 14 second-order GLDM features. The robustness (ICC) is above 0.8, per all values of distance and binw, only for nine of them. Robustness of many features across segmentations is dependent on bin-width and distance (the vertical dashed red line indicates the 0.9 threshold).

Figure 5. Robustness of second-order GLDM features across segmentations. Pyradiomics computes 14 second-order GLDM features. The robustness (ICC) is above 0.8, per all values of distance and binw, only for nine of them. Robustness of many features across segmentations is dependent on bin-width and distance (the vertical dashed red line indicates the 0.9 threshold).

Figure 6. Robustness of second-order GLRLM features across segmentations. Pyradiomics computes 16 second-order GLRLM features. The robustness (ICC) is above 0.8, per all values of distance and binw, only for 11 of them. Robustness of many features across segmentations is dependent on bin-width but not on distance (the vertical dashed red line indicates the 0.9 threshold).

Figure 6. Robustness of second-order GLRLM features across segmentations. Pyradiomics computes 16 second-order GLRLM features. The robustness (ICC) is above 0.8, per all values of distance and binw, only for 11 of them. Robustness of many features across segmentations is dependent on bin-width but not on distance (the vertical dashed red line indicates the 0.9 threshold).

Figure 7. Robustness of second-order GLSZM features across segmentations. Pyradiomics computes 16 second-order GLSZM features. The robustness (ICC) is above 0.8, per all values of distance and binw, only for 11 of them. Robustness of many features across segmentations is dependent on bin-width but not on distance (the vertical dashed red line indicates the 0.9 threshold).

Figure 7. Robustness of second-order GLSZM features across segmentations. Pyradiomics computes 16 second-order GLSZM features. The robustness (ICC) is above 0.8, per all values of distance and binw, only for 11 of them. Robustness of many features across segmentations is dependent on bin-width but not on distance (the vertical dashed red line indicates the 0.9 threshold).

Figure 8. Robustness of second-order NGTDM features across segmentations. Pyradiomics computes five second-order NGTDM features. The robustness (ICC) is above 0.9, for all values of distance and binw, only for one of them. Robustness of many features across segmentations is dependent on bin-width but not on distance (the vertical dashed red line indicates the 0.9 threshold).

Figure 8. Robustness of second-order NGTDM features across segmentations. Pyradiomics computes five second-order NGTDM features. The robustness (ICC) is above 0.9, for all values of distance and binw, only for one of them. Robustness of many features across segmentations is dependent on bin-width but not on distance (the vertical dashed red line indicates the 0.9 threshold).

Figure 9. Robustness of wavelet-based first-order histogram features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides 18 first-order features. The robustness (ICC) is above 0.8 only for Skewness, Kurtosis, Uniformity and 90Percentile. The latter has ICC > 0.90. They are not much dependent on bin-width (only entropy and uniformity have slight dependence). Because they are first-order, they are not dependent on distance neither do they depend on resolution or interpolator. However, the wavelet features are dependent on the direction used for wavelet computation (each circle on a line is a different gradient direction).

Figure 9. Robustness of wavelet-based first-order histogram features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides 18 first-order features. The robustness (ICC) is above 0.8 only for Skewness, Kurtosis, Uniformity and 90Percentile. The latter has ICC > 0.90. They are not much dependent on bin-width (only entropy and uniformity have slight dependence). Because they are first-order, they are not dependent on distance neither do they depend on resolution or interpolator. However, the wavelet features are dependent on the direction used for wavelet computation (each circle on a line is a different gradient direction).

Figure 10. Robustness of wavelet-based second-order GLCM features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides 24 such features. The robustness (ICC) is above 0.9 only for few of them. They are dependent on bin-width (binw: 10, 20, 40) and on (dst: 1, 2, 4). Because they are second-order, they are dependent on distance (dst) but they do not depend much on resolution or interpolator. However, the wavelet features are dependent on the direction used for wavelet computation (each circle on a line is a different gradient direction).

Figure 10. Robustness of wavelet-based second-order GLCM features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides 24 such features. The robustness (ICC) is above 0.9 only for few of them. They are dependent on bin-width (binw: 10, 20, 40) and on (dst: 1, 2, 4). Because they are second-order, they are dependent on distance (dst) but they do not depend much on resolution or interpolator. However, the wavelet features are dependent on the direction used for wavelet computation (each circle on a line is a different gradient direction).

Figure 11. Robustness of wavelet-based second-order GLDM features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides 14 such features. The robustness (ICC) is above 0.9 only for few of them. They are dependent on bin-width (binw: 10, 20, 40) and on (dst: 1, 2, 4). Because they are second-order, they are dependent on distance (dst) but they do not depend much on resolution or interpolator. However, the wavelet features are dependent on the direction used for wavelet computation (each circle on a line is a different gradient direction).

Figure 11. Robustness of wavelet-based second-order GLDM features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides 14 such features. The robustness (ICC) is above 0.9 only for few of them. They are dependent on bin-width (binw: 10, 20, 40) and on (dst: 1, 2, 4). Because they are second-order, they are dependent on distance (dst) but they do not depend much on resolution or interpolator. However, the wavelet features are dependent on the direction used for wavelet computation (each circle on a line is a different gradient direction).

Figure 12. Robustness of wavelet-based second-order GLRLM features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides 16 such features. The robustness (ICC) is above 0.9 only for few of them. They are dependent on bin-width (binw: 10, 20, 40) and on (dst: 1, 2, 4). Because they are second-order, they are dependent on distance (dst) but they do not depend much on resolution or interpolator. However, the wavelet features are dependent on the direction used for wavelet computation (each circle on a line is a different gradient direction).

Figure 12. Robustness of wavelet-based second-order GLRLM features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides 16 such features. The robustness (ICC) is above 0.9 only for few of them. They are dependent on bin-width (binw: 10, 20, 40) and on (dst: 1, 2, 4). Because they are second-order, they are dependent on distance (dst) but they do not depend much on resolution or interpolator. However, the wavelet features are dependent on the direction used for wavelet computation (each circle on a line is a different gradient direction).

Figure 13. Robustness of wavelet-based second-order GLSZM features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides 16 such features. The robustness (ICC) is above 0.9 only for few of them. They are dependent on bin-width (binw: 10, 20, 40) and on (dst: 1, 2, 4). Because they are second-order, they are dependent on distance (dst) but they do not depend much on resolution or interpolator. However, the wavelet features are dependent on the direction used for wavelet computation (each circle on a line is a different gradient direction).

Figure 13. Robustness of wavelet-based second-order GLSZM features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides 16 such features. The robustness (ICC) is above 0.9 only for few of them. They are dependent on bin-width (binw: 10, 20, 40) and on (dst: 1, 2, 4). Because they are second-order, they are dependent on distance (dst) but they do not depend much on resolution or interpolator. However, the wavelet features are dependent on the direction used for wavelet computation (each circle on a line is a different gradient direction).

Figure 14. Robustness of wavelet-based second-order NGTDM features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides five such features. The robustness (ICC) is above 0.9 only for few of them. They are dependent on bin-width (binw: 10, 20, 40) and on (dst: 1, 2, 4). Because they are second-order, they are dependent on distance (dst) but they do not depend much on resolution or interpolator. However, the wavelet features are dependent on the direction used for wavelet computation (each circle on a line is a different gradient direction).

Figure 14. Robustness of wavelet-based second-order NGTDM features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides five such features. The robustness (ICC) is above 0.9 only for few of them. They are dependent on bin-width (binw: 10, 20, 40) and on (dst: 1, 2, 4). Because they are second-order, they are dependent on distance (dst) but they do not depend much on resolution or interpolator. However, the wavelet features are dependent on the direction used for wavelet computation (each circle on a line is a different gradient direction).

Figure 15. Sensitivity to other parameters. Per each feature, the range of variation of ICC is reported per each parameter (bin-width, pixel distance, interpolator and isotropic resolution) having fixed the other three.

Figure 15. Sensitivity to other parameters. Per each feature, the range of variation of ICC is reported per each parameter (bin-width, pixel distance, interpolator and isotropic resolution) having fixed the other three.

Figure 16. Correlation between CV for each feature (on x-axis) and lesion size (shape feature Voxel Volume). Spearman method was used. Red circles indicate correlation for ‘shape’ features; only a weak correlation exists.

Figure 16. Correlation between CV for each feature (on x-axis) and lesion size (shape feature Voxel Volume). Spearman method was used. Red circles indicate correlation for ‘shape’ features; only a weak correlation exists.

Table 1. CT equipment and acquisition parameters.

Table 1. CT equipment and acquisition parameters.

Pixel spacing0.578–0.976 mmSlice thickness1.5 mmKVP120 kVEquipmentRevolution HD—GE MEDICAL SYSTEMSScan optionsHelical Mode

Table 2. Patient characteristics.

Table 2. Patient characteristics.

Sex14 F, 34 MAge49–87 (mean 70, std 10)Diagnosis 32 Adenocarcinoma,
14 Squamous,
1 Neuroendocrine
1 atypical carcinoid