記住我
ORIGINAL ARTICLE
Year : 2022 | Volume
: 18
| Issue : 7 | Page : 2006-2012
Comparison of nomogram with random survival forest for prediction of survival in patients with spindle cell carcinoma
Xiaoshuai Zhang1, Jing Liang2, Zhaohui Du3, Qi Xie4, Ting Li3, Fang Tang5
1 Department of Data Science, School of Statistics, Shandong University of Finance and Economics, Jinan, China
2 Department of Oncology, The First Affiliated Hospital of Shandong First Medical University, Jinan, China
3 Fred Hutchinson Cancer Research Center, Seattle, WA 98109, USA
4 Medical Research Center, The First Affiliated Hospital of Shandong First Medical University, Jinan, China
5 Center for Data Science in Health and Medicine, The First Affiliated Hospital of Shandong First Medical University; Shandong Qianfoshan Hospital, Cheeloo College of Medicine, Shandong University, Jinan, China
Correspondence Address:
Fang Tang
Center for Data Science in Health and Medicine, The First Affiliated Hospital of Shandong First Medical University, Jingshi Road 16766, 250014, Jinan
China
Source of Support: None, Conflict of Interest: None
CheckDOI: 10.4103/jcrt.jcrt_2375_21
Purpose: Spindle cell carcinoma (SpCC) is a relatively rare tumor with an unfavorable prognosis. This study aimed to develop and validate a prediction model for the individual survival of patients with SpCC using Cox regression and the random survival forest (RSF) model.
Methods: Patients diagnosed with SpCC between 2004 and 2016 were selected from the Surveillance, Epidemiology, and End Results (SEER) database, and randomly divided into training and validating cohorts. Cox regression and RSF were used to identify prognostic predictors and build prediction models. A nomogram based on Cox regression was constructed to predict the 1-, 3-, and 5-year survival of patients with SpCC. Internal validation was conducted using the bootstrapping method. We evaluated the discrimination accuracy and calibration of the model using Harrell's C-index and calibration plot, respectively.
Results: Two hundred and fifty patients diagnosed with SpCC with required information were enrolled in this study. Multivariate Cox regression and RSF identified age, primary site, grade, SEER stage, tumor size, and treatment as significant prognostic predictors of SpCC. The bootstrapped and validated C-indices were 0.812 and 0.783 for nomogram, and 0.790 and 0.768 for RSF, respectively. Calibration plot of the nomogram showed an agreement between the prediction and actual observation.
Conclusions: The nomogram developed in this study is a promising tool with a simplified presentation that can easily be used and interpreted by clinicians for evaluating the survival of each patient with SpCC; its performance was comparable to that of RSF. Application of such models are needed to help oncologists identify the high-risk patients and improve clinical decision making of SpCC treatment.
Keywords: Nomogram, prognosis, random survival forest, SpCC, survival prediction
Spindle cell carcinoma (SpCC) is a relatively rare cancer with high recurrence and metastasis rates.[1],[2],[3] Due to the rarity of these tumors, the prevalence or incidence rates of SpCC among the population are barely assessed. Previous research has reported a peak incidence of SpCC in the seventh decade of life without gender difference based on the Surveillance, Epidemiology, and End Results (SEER) database.[4] However, much of the remaining literature is based on studies with limited sample sizes or case reports, and robust conclusions about the treatment modalities and standard treatment protocol in SpCC could not be drawn.[5],[6] In addition, it has been suggested that the mainstay treatment for SpCC is surgery, and the effects of adjuvant radiotherapy or chemoradiotherapy varied based on tumor location.[7],[8],[9],[10] Patients with SpCC usually have unfavorable prognoses and management strategies that vary a cross previous studies.[2],[5] Although a few studies have indicated that demographic characteristics, clinicopathologic features, and treatment modalities were associated with prognosis, the established predictors of SpCC survival were still limited.[5],[11] Therefore, to improve the prognosis of SpCC, it is necessary to build instructive prediction tools to better support the clinical decision-making process.
Nomogram is a practical decision-making tool for clinicians, which has successfully been used to predict the probability of a clinical outcome for a specific patient.[12],[13],[14],[15],[16],[17],[18] A nomogram is the graphical representation of a statistical predictive model such as logistic and Cox proportional hazards models. Machine learning methods have shown higher accuracy in classification or prediction than traditional methods. Random survival forest (RSF) is a nonparametric algorithm used for analyzing right-censored survival data, and the promising performance of RSF has been illustrated in different studies.[19],[20],[21]
Notably, to the best of our knowledge, there has been no attempt to use a nomogram or machine learning methods for SpCC prediction. Therefore, this study developed and validated a risk prediction nomogram to predict the survival of patients with SpCC using the SEER database. Moreover, RSF was conducted to compare the performance with that of the proposed nomogram.
> Material and MethodsPatient selection
Patients diagnosed with SpCC from 2004 to 2016 were identified using the Incidence-SEER 18 Regs Custom Data with additional treatment fields, Nov2018 Sub (1975–2016 varying). Patients with SpCC were searched for using SpCC, NOS (8032/3), according to the third edition of the International Classification of Diseases for Oncology. Deaths attributed to SpCC were defined as end events, and being alive or dead from other causes were defined as censored observations. The exclusion criteria were as follows: patients with survival time less than one month or unknown; patients diagnosed in 2016 with follow-up time less than one year; and patients with multiple primaries, no or unknown treatment, unknown stage, unknown grade, or unknown tumor size. Two hundred and fifty participants were eligible for this study.
The following demographic and clinicopathological variables were examined in this study: age, sex, race, primary site, SEER summary stage, pathological grade, tumor size, and treatment. The SEER summary stage 2000 was used to classify the neoplasm stages as localized, regional, and distant. For the pathological grade, moderately differentiated and well differentiated were combined as low grade, and undifferentiated and poorly differentiated were combined as high grade. Tumor sizes were categorized as ≤5 cm and >5 cm. The treatment modality was classified as surgery alone, chemotherapy alone, radiation alone, surgery + chemotherapy, surgery + radiation, chemotherapy + radiation, and surgery + chemotherapy + radiation. Primary tumor sites were classified into six categories: head and neck, chest, breast, alimentary system, urinary system, and other organs. The Ethics Committee of The First Affiliated Hospital of Shandong First Medical University exempted this study since SEER research data are publicly available and deidentified.
Statistical analysis
Baseline characteristics of patients with SpCC by survival status were described as means (standard deviation, SD) for continuous variables and proportions for categorical variables. For prediction model construction and validation, 2/3 of the patients were randomly assigned to the training cohort and 1/3 to the validation cohort. Univariate and multivariate Cox proportional hazards models were used to identify potential predictors, and variables with P < 0.05 were included in the multivariate Cox model. Then, a nomogram was constructed for predicting the 1-, 3-, and 5-year survival rates of patients with SpCC based on the established prediction model. The nomogram was validated internally in the training cohort and then in the validation cohort. Harrell's C-index was used to evaluate the prediction model's discrimination accuracy.[22] A calibration plot was generated to assess the agreement between the survival rate predicted by the nomogram and the observed survival rate. Internal validation of the proposed model was conducted using the bootstrapping method. Internal validated C-index was calculated for the training cohort by subjecting the nomogram to 1000 bootstrap resamples, and calibration was assessed by comparing the overfitting-corrected predicted and observed probability with 1000 bootstrap resamples.
In addition, RSF was used to predict SpCC survival compared with the nomogram. RSF is an assembly of decision trees using bootstrap samples designed for right-censored time-to-event data. Each bootstrap sample consists of approximately 63% of the observations, and the remaining 37% as the Out-of-Bag sample can be used to validate the model. The variable importance (VIMP) algorithm was used for variable selection.[23] For a given variable, the VIMP is the difference between the prediction error under the observed value and that when the variable is randomly permuted. Positive VIMP indicates that the variable improves the predictive accuracy, and zero or negative VIMP shows that the variable has no or adverse effect on the prediction. Variables with positive VIMP values were selected, and those with negative VIMP values were removed from the final prediction model. All statistical analyses were conducted using R v. 3.6.1 with the “rms” and “random Forest SRC” packages.
> ResultsCharacteristics of the patients
Two hundred and fifty participants were included in this study, among which 109 (43.6%) were male and 141 (56.4%) were female. The mean age of all patients was 65.76 (SD: 13.66) years. The median follow-up period was 1.17 years. The training and validation cohorts consisted of 166 and 84 cases, respectively. The mean age of patients was 65.37 (SD: 14.72) years in training cohort, and 66.53 (SD: 11.33) years in validation cohort. [Table 1] summarizes the baseline demographic and clinicopathologic characteristics of patients in the training and validation cohorts.
Cox regression models
[Table 2] shows the results of the univariate and multivariate Cox models in the training cohort. Age, sex, primary site, grade, stage, tumor size, and treatment were statistically significantly associated with survival in the univariate analyses and were therefore included in the multivariate Cox proportional hazards model. The multivariate model identified six significant predictors of SpCC survival, including age, primary site, grade, stage, tumor size, and treatment.
Table 2: Univariate and multivariate Cox proportional hazards regression analysisNomogram and validation
[Figure 1] shows the nomogram incorporating the six significant risk factors for predicting the 1-, 3-, and 5-year survival rates of the patients with SpCC. The nomogram gives a score for each variable, and the total point of one patient was obtained by summing the scores of all variables. Then, the 1-, 3-, and 5-year survival rates of one patient can be estimated with a vertical line dropping down from the total point. In the training cohort, the original C-index of the nomogram was 0.840 (95% CI, 0.798–0.861) and the bootstrapping C-index was 0.812 (95% CI, 0.762–0.845). The C-index in the validation cohort was 0.783. [Figure 2] displays the calibration plots of the 1-, 3-, and 5-year survival probabilities after treatment in the training and validation cohorts. Again, the calibration plot slopes for the nomogram were close to 1, showing good agreements between predicted survival probability and the actual observations, especially for the 3-year prediction.
Figure 1: Nomogram predicting 1-, 3-, and 5-year survival of the patients with SpCC after treatment. Abbreviations: H-N, Head and neck; AS, Alimentary system; US, Urinary system; S, Surgery; C, Chemotherapy; RT, Radiation; Stage, SEER summary stage; Grade, pathological grade
Figures 2: Calibration plot for predicting the survival of patients with SpCC. (a–c) shows the 1-, 3-, and5-year endpoints in the training cohort, and (d–f) shows the 1-, 3-, and 5-year endpoints in the validation cohortPrediction model with RSF
[Figure 3] indicates the VIMP of each variable and the prediction error of RSF in the training cohort. Similar to the Cox regression, all variables except sex and race have positive VIMP, indicating that those variables contribute to the predictive power of the RSF model. The most predictive variable is stage, followed by treatment, primary site, tumor size, grade, and age [Figure 3]a. [Figure 3]b displays the prediction error after removing sex and race from the model. The prediction error decreases with the number of trees and becomes stable when the number reaches 2000. The bootstrap C-statistics of the RSF prediction model are 0.790 (95% CI, 0.721–0.810) in the training cohort, and 0.768 in the validation cohort, respectively. The marginal effect of each variable is shown in the supplementary file S1.
Figure 3: VIMP values of each variable and prediction error of RSF. (a) shows the VIMP values of each variable, and (b) depicts the prediction error rates for RSF after variable selection > Discussion Using the SEER database in this study, we first developed a nomogram to illustrate the survival of patients with SpCC in a straight forward way for clinical usage. We subsequently compared the prediction performance of the nomogram with that of a machine learning method, RSF.
SpCC is a rare and aggressive disease that presents heterogeneous pathological features, clinical behavior, and prognosis. It is informative to establish a prediction model using personal disease characteristics to estimate individualized survival rates, which may help physicians and patients in decision-making. However, previous studies on SpCC survival only used Kaplan–Meier estimates to calculate disease-specific survival rates for the SpCC population,[24],[25],[26],[27],[28] and rarely has any study attempted to establish prediction models from the individual level. The nomogram developed in this study was a promising tool with a simplified presentation that clinicians could easily use to evaluate the survival of each patient with SpCC. Nomograms have been successfully applied to predict survival in patients with lung, breast, and other cancers.[29],[30],[31],[32] A nomogram depicts the results of Cox regression within a graph. In the nomogram, the effect of each variable was assigned as a point score represented on the X-axis, and the cumulative score for all variables of an individual was matched to the survival probability at the specific time point. Besides, all predictors used in our proposed nomogram are common clinicopathological variables that can be easily acquired from patients, indicating that this nomogram can be widely applied in clinical practice.
We examined the discrimination and calibration ability of the nomogram using the C-index and calibration plot, respectively. Bootstrapping is an effective internal validation method used to produce unbiased estimates and help avoid overfitting.[33] The C-index decreased from 0.812 to 0.783 in the validation cohort demonstrating an acceptable discriminative performance. In the calibration plot, the survival probabilities predicted by the nomogram were close to the observed probabilities, indicating a good calibration performance.
Several machine learning algorithms have been increasingly used in disease prediction and have shown superior performance compared with conventional statistical methods.[34],[35],[36] Thus, we compared the nomogram with RSF, a random forest method for the analysis of right-censored survival data. Previous studies have shown that RSF is better than or at least comparable to Cox regression with respect to prediction performance.[20],[37],[38] In this study, the two models had comparable performance, and the point estimate of the C-index of the nomogram was slightly higher than that of RSF. One possible reason was that the number of variables was small and RSF failed to show the advantages. Additionally, the nomogram requires less computational resources than machine learning methods, and it is a simple and user-friendly method that generates the probability of an outcome for a given individual. Therefore, the nomogram may be more appropriate in clinical practice.
The nomogram identified stage, treatment, and primary site as the top three predictors, which also had high VIMP in RSF. Stage and grade contributed substantially to the prediction since patients with advanced cancer were more likely to have unfavorable outcomes. Feng et al.[4] reported that the surgery alone group had better survival than the radiotherapy alone and surgery + radiotherapy groups using univariate analysis. In the univariate analysis, patients who received surgery alone had significantly better survival than patients who received the other four types of treatment, including radiation alone, chemotherapy alone, chemotherapy + radiation, and surgery + radiation. However, after adjusting for grade, stage, primary site, and other covariates, surgery alone only had better survival than radiation alone. RSF also indicated that patients with radiotherapy alone had the lowest survival probability (see the supplementary file). Although an observational study is not a preferred method to evaluate the effect of a treatment since it may be biased due to unknown confounders, the treatment can still be a useful predictor in estimating the probability of SpCC survival. The effectiveness of the treatment modality needs further validation in clinical trials. Most previous studies on SpCC mainly focused on identifying prognosis factors for a specific type of SpCC. We also investigated the effect of different primary sites on the survival of patients with SpCC. The primary site was identified as the third most important predictor by the nomogram and RSF. Compared with other organs, the urinary system had the worst prognosis, followed by the chest, head and neck, alimentary system, and breast. Tumor size >5 cm was a negative predictor of SpCC survival, which has been reported as a prognosis factor for head and neck SpCC.[5] It was reported that increasing age could be a risk factor for the survival of patients with SpCC.[4],[5] In this study, gender was significantly associated with survival in the univariate Cox model but not in the multivariate analysis, which was consistent with the results of a previous study.[6]
This study has several limitations. First, the data were retrospectively collected from the SEER dataset, and the sample size was relatively small. Stratified analysis with an adequate sample size could help improve the performance of the prediction. Second, the models were validated only among the homogeneous populations due to the lack of an external cohort. Third, more appropriate models such as competing risk models should be considered to improve the prediction ability in further work.
> ConclusionsSpCC is a rare disease with an unfavorable prognosis, while reliable prediction models quantifying the survival probability for individual patients are still lacking. We developed a simplified prediction model that included easily accessible factors for oncologists and generated good performance for practical use. Both demographic and cancer-related features, including age, primary site, grade, stage, tumor size, and treatment modality, were informative in predicting the survival probabilities for patients with SpCC. Although the performance of the nomogram and RSF was comparable, the nomogram can easily be used and interpreted by clinicians in clinical practice. The validation results demonstrated that this nomogram had good discrimination accuracy and calibration. Such models are needed to help oncologists identify high-risk patients and to improve the clinical decision-making in SpCC treatment.
Financial support and sponsorship
This work was supported by National Natural Science Foundation of China (81903410 and 71804093), Academic Promotion Programme of Shandong First Medical University (2019LJ005 and 2019QL013).
Conflicts of interest
There are no conflicts of interest.
> References
留言 (0)