WHAT IS ALREADY KNOWN ON THIS TOPIC

Dialysis patients have a high mortality rate with wide interindividual variability in prognosis.

Identifying patients at high risk for early death is necessary to facilitate the planning and delivery of timely supportive care services.

There are currently no models to predict death for prevalent dialysis patients that have evidence for external validity or that have been implemented and evaluated in routine clinical care.

WHAT THIS STUDY ADDS

This study provides evidence for external validity of an integrated model to predict short-term and long-term mortality risk in prevalent haemodialysis patients when integrated into routine care at a programme level.

Results showed feasibility of implementation of the Cohen prognostic model in routine dialysis care at a programme level.

Although the model tended to overpredict mortality risk, especially with long-term predictions of 12 months and 18 months, it showed good discrimination between high-risk and low-risk groups.

HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY

The Cohen prognostic model can be used in routine clinical practice to identify high-risk haemodialysis patients who can be targeted for supportive care interventions.

Discriminating patients at high risk for early mortality may also aid in resource planning and aid in research through better defining study enrolment criteria nearer the end of life.

There are numerous prognostic models in the literature, but they cannot be recommended for clinical use without evidence for external validity or clinical utility and feasibility.

To prevent waste of a large body of research, external validation studies such as this are required of other prognostic models for both incident and prevalent patients across diverse dialysis populations.

These prognostic models should also extend to other outcomes of importance to people undergoing dialysis in addition to survival.

Introduction

Patients on maintenance haemodialysis have high annual mortality rates and complex care needs. Their illness trajectories appear particularly variable;1 2 predicting these trajectories and communicating prognosis is integral to quality care. International guidelines explicitly recommend communicating prognosis to facilitate shared decision-making; align care plans with patients’ personal preferences, needs and goals of care; and aid timely and effective planning of supportive care services.3 4 Even though most patients want to know their prognosis and future illness trajectories,5–8 evidence suggests that sharing this information is not yet a standard clinical practice5 9 10 and patients often receive high treatment intensity at the end of life despite a poor prognosis and often in poor alignment with their preferences.5 11

Although several prognostic systems in the form of statistical models have been proposed for patients receiving dialysis, the number of studies that attempt to combine prognostic factors into clinically useful prediction tools is small with most focusing on incident patients.12 13 Current models predicting death have limited if any evidence for external validity and none have been implemented and evaluated in clinical practice on a large scale.14

The need to develop prognostic models that can be integrated at the bedside to guide clinical care is a high priority in nephrology and has been incorporated into the International Society of Nephrology’s recently developed Strategic Plan for Integrated Care of Patients with Kidney Failure.15 To date, we are unaware of any studies that have evaluated the performance or utility of prognostic tools in real life clinical practice for prevalent haemodialysis patients.

Our overall aim is to develop a clinical standard to identify patients on haemodialysis who are at high risk for death within the next 6 months, 12 months and 18 months. This study specifically sought to provide evidence for external validity in a real life setting of the Cohen prognostic model that integrates actuarial factors with clinician predictions to assess mortality risk of prevalent haemodialysis patients.16 Models that combine clinician predictions with actuarial factors have been shown to have better prognostic accuracy in predicting survival in patients nearing the end of life.17

Methods

In 2010, Alberta Kidney Care-North, a Canadian university-based programme, introduced a supportive care assessment for all patients on dialysis, which included twelve incentre haemodialysis units.18 19 The goal was to prospectively identify patients with high care needs who may benefit from additional supportive or palliative care services. This involved routine screening of various indicators, including the Cohen integrated prognostic model for mortality risk, the Edmonton Symptom Assessment System-revised:Renal for symptom burden, and the modified Karnofsky and activities of daily living to assess functional status.20 21 All assessments were recorded in the local electronic medical record, which also provides laboratory data, demographic characteristics, comorbidity and dialysis-related information (modality, cause of ESKD (end stage kidney disease), vascular access, and dialysis start and end dates). Data were collected every 3 months between August 2010 and October 2019. Patients who had been receiving maintenance haemodialysis for ≥3 months with at least 6 months of follow-up information were eligible for inclusion into the validation cohort.

The first assessment date was used to calculate the predictive risk index based on the Cohen model, with predictors: ‘Surprise Question’—‘Would you be surprised if this patient were to die in the next 12 months?’, age, serum albumin, dementia and peripheral vascular disease. Assessment dates ranged from May 2011 to April 2019. For patients who did not have an albumin result associated with the assessment where other information was collected, the last available lab result was used. We used the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis recommendations for ideal reporting of the validation of the prognosis model.

All statistical analysis was done using R V.4.0.4. Patient characteristics were presented as counts (%) for categorical variables, and medians (IQR) or means (SD) for continuous variables. Differences between the Cohen derivation data and the external validation data were tested with t-tests for continuous variables and χ2 tests for categorical variables. The predicted risk index and associated predicted survival for 6 months, 12 months and 18 months were calculated using the Cohen model. Follow-up times were censored administratively at 24 months to match the derivation cohort.

Observed survival was plotted as Kaplain-Meier curves, and as bar plots alongside mean predicted survival for each of the time points estimated. Risk groups were calculated based on the predicted risk index scores generated in the validation data set by both quintiles (in keeping with the Cohen approach), as well as by using Cox-based cut-offs at the 16th, 50th and 84th centiles. These cut-offs have been recommended as they minimise loss of information in the grouping process.22 23 Kaplain-Meier curves by risk group strata were plotted for both approaches, and HRs with 95% CI were calculated to assess the differences between risk group survival.

Calibration and discrimination of the Cohen model were assessed by calculating the cumulative area under the curve (AUC) at each time point and Somer’s Dxy. Additionally, a calibration slope was calculated using the predicted risk index with likelihood ratio test against the hypothesis that the slope is equal to 1. A joint test on all Cohen predictor variables after offsetting the predicted risk index was run to examine any sources of miscalibration.

New baseline survival estimates for the validation data set were calculated based on both a Cox Proportional Hazard and a parametrical Weibull approach. The updated baseline survival functions were combined with the Cohen predicted risk index to estimate updated predicted survival, and were compared against the observed survival as well as the original Cohen mean predicted survival at each time point.

Results

A total of 1372 patients was included in the validation cohort. Of these, 284 (26.1%) died within the 24-month follow-up period. Demographic and clinical characteristics of the validation cohort are described in table 1.

Table 1

Patient characteristics

There were many differences between the validation cohort and the Cohen derivation cohort (table 2). Notably, our validation cohort had a much higher proportion of ‘not surprised’ responses to the surprise question (41% vs 15.8%, p<0.0001), lower serum albumin levels (mean 3.6 vs 3.8 g/dL, and 36.3% vs 16.9% with <3.5 g/dL, p<0.0001), higher rates of peripheral vascular disease (17.4% vs 3.3%, p<0.0001) and lower rates of dementia (2.4% vs 19.3%, p<0.0001). In addition, 18.7% of the Cohen derivation cohort were incident patients with less than 3 months on hemodialysis (HD).

Table 2

Differences in patient characteristics between the Cohen derviation and the external validation cohorts

The Kaplan-Meier (KM) survival curve for the validation cohort is shown in figure 1, along with the 6-month, 12-month and 18-month observed versus predicted survivals. In the validation cohort, the Cohen model predicted outcomes well in the early time points, but overpredicted death further out. The difference between observed survival and mean predicted survival at 6 months, 12 months and 18 months was 3.2%, 8.8% and 12.9%, respectively. The discrimination of the model in the validation cohort was moderate, with 6-month, 12-month and 18-month AUCs of 0.72, 0.71 and 0.71, respectively (figure 2). The Somer’s Dxy Score was 0.387 and the calibration slope was 0.598 (SE 0.05, p<0.0001) suggesting worse discrimination than the Cohen derivation data. A joint test of the Cohen predictors after offsetting the predictive index showed evidence that the coefficients were not equal to 0 (p<0.0001). Specifically, there appeared to be large differences in the surprise question and albumin results between the derivation and validation cohorts that may impact predictive accuracy.

Figure 1

Kaplan-Meier curve for the validation cohort with observed versus predicted mean survivals at 6 months, 12 months and 18 months.

Figure 2

ROC (receiver operating characteristic) curves with cumulative AUC at 6 months, 12 months and 18 months.

Kaplan-Meier curves stratified by both quintile-based and Cox-based risk groups showed good discrimination between groups, although there was some overlap in groups 2, 3 and 4 in the quintile approach (figure 3A). The HR estimates for each group compared with the lowest-risk group suggested the model can assign higher risk scores to patients who did have worse outcomes (quintile HRs: Q2 vs Q1 2.81, Q3 vs Q1 4.76, Q4 vs Q1 5.71, Q5 vs Q1 10.64; Cox HRs: C2 vs C1 3.07, C3 vs C1 5.85, C4 vs C1 13.24; figure 3A,B).

Figure 3

(A) Kaplan-Meier curves with HRs and 95% CIs for Predicted Risk Index quintiles. (B) Kaplan-Meier curves with HRs and 95% CIs for Cox-based risk groups.

The observed versus mean predicted survival for 6 months, 12 months and 18 months were calculated by predicted risk quintile (figure 4). While there appears to be good agreement in the lower-risk groups at all time points, quintiles 4 and 5 have widening gaps between observed patient survival and what is predicted by the Cohen model, especially at the later time points. In particular, the highest risk group (Q5) suggests that the Cohen model is overestimating poor outcomes for high-risk patients by a large margin (70%, 44% and 27% mean predicted 6-month, 12-month and 18-month survival vs 84%, 75% and 65% observed).

Figure 4

Observed versus predicted mean survival at 6 months, 12 months and 18 months by quintile.

After estimating baseline survival at 6 months, 12 months and 18 months based on both Cox Proportional Hazard (PH) and Weibull models, the updated predicted mean survival at each time point was higher than the Cohen estimates, and in fact survival was overestimated at each time point (online supplemental figure S1).

Discussion

Nephrologists have a moral and professional obligation to discuss prognosis to promote informed shared decision-making and optimise care.24 However, this requires accurate prognostication using tools that can be integrated into routine clinical care. Unfortunately, most of the current prognostic tools are limited to survival predictions of incident dialysis patients. Scores may not be easy to calculate or may use data that are not collected routinely. There is also a lack of information on predictive performance of scores or risk thresholds to inform clinical practice and most tools lack external validation.25 As a result, the clinical utility and usability for many of the tools are limited, hindering meaningful translation to the bedside, and leaving nephrologists wary of sharing uncertain prognostic information.

Our study provided evidence for external validity for 6-month, 12-month and 18-month survival predictions for prevalent haemodialysis patients using a simple integrated prognostic model that was implemented as part of routine dialysis care in our renal programme. To date, we are unaware of any prospective studies testing the performance or utility of a prognostic tool for prevalent haemodialysis patients in routine clinical practice.

Discrimination performance of the Cohen prognostic model was moderate with AUCs of 0.71–0.72 for all three time points. The model tended to overpredict mortality risk at all time points with the best predictive accuracy for 6-month survival. Discrimination performance was not as high in this external validation cohort compared with Cohen’s derivation and internal validation cohorts where AUCs for 6-month survival predictions were 0.87 (95% CI 0.82 to 0.92) and 0.80 (95% CI 0.73 to 0.88), respectively.12 This is the first time discrimination performance has been described for the 12-month and 18-month predictions for this tool. It is likely that the significant differences between the two patient cohorts impacted model accuracy. Compared with the derivation cohort, no incident patients were included in our validation cohort. The rates of peripheral vascular disease and a ‘not surprised’ response to the surprise question were surprisingly low in the derivation cohort (3.3% and 15.8%). On average, our patients appeared sicker with lower serum albumin, higher rates of vascular disease and a much higher proportion of ‘not surprised’ responses to the surprise question. The model overpredicted the impact these variables had on mortality.

The model, however, was good at discriminating between high-risk and low-risk patients, suggesting that this is a useful tool for identification of patients who might benefit from additional supportive care services including prioritising goals of care conversations and advance care planning. Clinical tools that can identify patients at high risk for early mortality also aid in resource planning and are extremely helpful in research such as better defining study enrolment criteria.

Our results confirm that we must continue to use this tool with caution, especially when incorporating results into shared decision-making and the development of care plans. While it can separate high-risk and low-risk patients relatively well, it remains imprecise when applied to the individual. For example, 50% of patients characterised as high risk will still do well. Therefore, their use to restrict treatment options such as dialysis or other services remains problematic; while we do not want to be overoptimistic in the face of a poor prognosis, we also do not want to narrow patient choice or withdraw treatment based on overprediction of death leading to a self-fulfilling prophecy. We need to appreciate the uncertainty of the estimates and the possibility of a change in status while considering the individual patient context.

Our study has limitations. Although we had an ethnically diverse cohort, it was predominantly Caucasian. We also had an extremely low rate of recognised dementia, which most likely reflects underdetection.12 26 Formal screening for cognitive impairment was not undertaken; incorporation of cognitive screening and recognition of dementia may have improved the tool’s performance. Strengths include a large sample size over many years in a real life clinical setting. Our cohort also included patients with different baseline characteristics from the derivation cohort, increasing generalisability. The tool was incorporated easily into routine clinical care, showing overall feasibility, acceptability and clinical utility.

In conclusion, the Cohen integrated prognostic model is a simple clinical tool that can be incorporated easily into routine dialysis care to identify prevalent haemodialysis patients who are at high risk for death over 6 months, 12 months and 18 months. It performs best at earlier time points. While this tool is not sensitive enough to tell a patient how long they will live, it is informative at identifying high-risk patients who can then be targeted for specific interventions that might potentially benefit them.

Further work into survival estimation should be done to include different patient types, including peritoneal dialysis patients. We also need to improve model performance. Functional or nutritional decline or decreased quality of life scores may signal shortened survival, acting as sentinel events that can be readily observed and measured and may increase predictive accuracy.27–29 We also need to better understand what risk factors are modifiable, for example, frailty.30 Next steps should include testing the effect of mortality predictions on shared decision-making.

Data availability statement

No data are available.

Ethics statementsPatient consent for publication

Not applicable.