Prognostic models are often used to estimate the length of patient survival. The Cox proportional hazards model has traditionally been applied to assess the accuracy of prognostic models. However, it may be suboptimal due to the inflexibility to model the baseline survival function and when the proportional hazards assumption is violated. The aim of this study was to use internal validation to compare the predictive power of a flexible Royston-Parmar family of survival functions with the Cox proportional hazards model. We applied the Palliative Performance Scale on a dataset of 590 hospice patients at the time of hospice admission. The retrospective data were obtained from the Lifepath Hospice and Palliative Care center in Hillsborough County, Florida, USA. The criteria used to evaluate and compare the models' predictive performance were the explained variation statistic R2, scaled Brier score, and the discrimination slope. The explained variation statistic demonstrated that overall the Royston-Parmar family of survival functions provided a better fit (R2 = 0.298; 95% CI: 0.236–0.358) than the Cox model (R2 = 0.156; 95% CI: 0.111–0.203). The scaled Brier scores and discrimination slopes were consistently higher under the Royston-Parmar model. Researchers involved in prognosticating patient survival are encouraged to consider the Royston-Parmar model as an alternative to Cox.
Citation: Miladinovic B, Kumar A, Mhaskar R, Kim S, Schonwetter R, et al. (2012) A Flexible Alternative to the Cox Proportional Hazards Model for Assessing the Prognostic Accuracy of Hospice Patient Survival. PLoS ONE 7(10): e47804. doi:10.1371/journal.pone.0047804
Editor: Raya Khanin, Memorial Sloan Kettering Cancer Center, United States of America
Received: May 11, 2012; Accepted: September 21, 2012; Published: October 17, 2012
Copyright: © Miladinovic et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: This study was supported by the United States Army Medical Research and Material Command grant DOA W81 XWH-09-0175. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Prognostic models are often used to estimate the length of patient survival and improvement in the accuracy of prognosis translates into superior quality of patient care. Precise prognosis of survival using modeling techniques requires rigorous methods for the development and testing of the accuracy of prognostic models. Developing a prognostic model entails having accurate patient data for prognosis, and selecting clinically relevant candidate predictors and measures of model performance, usually in the context of multivariable regression . This process produces patient performance scores that allow for classification of patients into different risk groups , , .
In the hospice setting, accurate prognostication of survival affords patients and their families a vital opportunity to attend to matters such as planning, prioritizing, and preparing for death . Predicting patient survival is a complex decision making process involving numerous subjective and numerical factors that have substantial variation which may lead to poor prediction of life expectancy. Many physicians practice optimism or avoidance, thus overestimating survival at times by a factor of five . Implementing appropriate statistical methodologies translates into improved accuracy of prognosis and superior quality of care. Predictions based on appropriate statistical modeling have been shown to be superior to physicians' prognostication , .
The Cox proportional hazards (CPH) model  is the most commonly-used survival prediction model , . In the hospice and palliative settings, demographic and clinical covariates are often included in CPH to predict patient survival , . The appeal of the model is its analytic simplicity and that the baseline survival function does not need to be defined apriori–it is absorbed when the likelihood function is maximized (note that “baseline” refers to zero values of the covariates, not to time equal to zero). It is possible to estimate the baseline survival function for the CPH model conditional on the estimated regression coefficients. However, this is highly rigid as the smoothing of the underlying function depends on the proportional hazards assumption, which may not be supported by the data and is often overlooked by the investigators . Essentially, the CPH model was designed to measure the effects of covariates on the changing hazard function and not to model patient survival. A flexible family of functions which allows for parametrically modeling the baseline survival function is more appropriate, especially if the proportional hazards assumption is violated in the CPH . The baseline survival has for the most part been ignored because it is left undefined in the CPH model.
In this manuscript we compare CPH with an alternative method of estimating survival in the form of the class of flexible Royston-Parmar (RP) parametric functions . We use the Palliative Performance Scale (PPS)  from a cohort of hospice patients. Results from systematic reviews have shown that the patient PPS score is an accurate measure of patient survival in the palliative setting , . Furthermore, PPS and CPH model have been used to construct meaningful hospice patient survival estimates in the form of a life expectancy table and survival nomogram  and to validate prognosticating scales for hospice patient survival , , .
In addition to PPS, other risk factors such as age, cancer status and gender have been reported to be significant predictors of palliative patient survival in several studies , ._ENREF_18 In our study we did not adjust for other risk factors because though they may be significant predictors of survival for the cohort of patients in our dataset, they may not be in other palliative settings. Our goal was to demonstrate that the RP family of parametric functions allowed for a direct and flexible modeling of the baseline survival and that it might be formulated so that the impact of the proportional hazard assumption is minimized. We determined if the overall performance and discriminatory ability of RP family of parametric functions is superior to CPH in the sample by using models that were derived and tested on the whole dataset (naïve validation) and using (internal) cross-validation. It is important to note that the RP parametric functions have not been applied to prognostic models in the hospice and palliative settings. It is also important to note that we did not perform external validation, which entailed using a different data set than the one used to create the model. In the next section we briefly discuss PPS, introduce the statistical models and measures of model performance.
Study sample and palliative performance
The patient data were obtained from the Lifepath Hospice and Palliative Care Center licensed since 1983 to serve in Hillsborough County, Florida. Hospice care focuses on pain control and symptom management. To avoid selection bias, we retrospectively and sequentially extracted data for 590 patients who, as of January 2009 were deceased. This study was a retrospective review of the deceased patients' medical records. Only data pertaining to outcomes were collected; personal information was not collected and the data were de-identified prior to analysis. Since we did not collect any information that can identify deceased patients or their family members, under HIPPA rules and regulations (45 CFR 164.512) the requirement for consent does not apply. The study and consent procedures were approved by the University of South Florida Institutional Review Board prior to study initiation. Two research assistants extracted all data necessary to populate the model variables and two faculty members randomly checked 25% of the data for accuracy. The models were tested against observed survival duration.
The Palliative Performance Scale (PPS) was developed and reported by Anderson et al.  as a measure of palliative patients' functional status. The scale has 11 possible mutually exclusive levels, which are based on five domains: six levels of ambulation, six levels of activity and evidence of disease, five levels of self-care, five levels of food intake and four levels of consciousness. The scale ranges from PPS of 0% (deceased patient) to PPS of 100% (ambulatory and healthy patient). Numerous studies have studied its prognostic accuracy of survival in a variety of settings and found it provides meaningful estimates of patient survival , , , , , , , , . PPS has been found to be both valid and reliable .
Model selection and validation
Validating a prognostic model is generally accepted to mean that given a patient population it works in a data set other than the one it is applied to, . In other words, the model needs to be tested using a different data set than the one used to create the model. It is also generally accepted that the validation process should follow guidelines and that un-validated prognostic models should not be applied in clinical practice , , . When validating a prognostic survival model in the regression framework, most attention has been on the value of the prognostic index based on covariates, while the role of the baseline survival function has been largely ignored.
The role of the baseline survival is significant as it quantifies the absolute patient survival probabilities over time. For a vector of covariates x and parameter vector β, the survival function S(t; x) at time t for the CPH model is commonly expressed as , where S0 (t) is the baseline survival function, i.e. survival function when all the covariates x are equal to zero. In the CPH framework, the estimation of the prognostic index xβ does not require the formulation of the baseline cumulative survival function S0 (t), which itself can be estimated conditional on the covariate estimates. The two popular methods for estimating baseline survival S0 (t) are the Breslow and Kalbfleisch-Prentice methods . Both give similar results in practice, but can lead to “choppy” estimates of the baseline function and are dependent on the proportional hazards assumption.
When the goal of a survival analysis is to estimate hazard ratios (the effect of covariates on the changing hazard function), the baseline function is of no consequence. The CPH is appropriate as the baseline function gets absorbed when coefficient βs are estimates by the method of partial log likelihood. However, when the goal is to prognosticate patient survival, there is a need for more flexibility in modeling the baseline survival.
An alternative to the CPH is the RP family of models that resembles the generalized linear models and can be viewed as a parametric extension Cox proportional hazard models . The models are framed to rely on the transformation g(.), such that . The transformation g(.) can be either from the proportional hazard, proportional odds, Aranda-Ordaz or probit families . We did not consider the Aranda-Ordaz family in this study due to possible interpretational difficulties . Under the proportional hazard link function, the hazard ratio estimates are nearly identical to those estimated under CPH. The attractive feature of the RP baseline survival function is that its shape is preserved, but the location of the baseline distribution function can vary, which allows for flexible model recalibration. Also, the estimate is implemented on log-time scale. It is generally gently curved and smooth, making survival estimates more accurate.
In the RP framework, if the proportional hazard assumption is violated, the probit-link function g(s) = −Φ−1(s) can be applied, where Φ−1(.) is the inverse standard normal distribution function. The baseline survival function is approximated and smoothed by a restricted cubic spline function with m interior knots. Splines are piecewise polynomials that ensure the overall curve is smooth (see Royston and Parmar  for details). Spline-based survival models such as RP have been empirically shown to be superior when the proportional hazard assumption is violated . The optimal number of knots and the comparison among different RP models can be found using the minimum combination of Akaike Information Criterion (AIC), Bayes Information Criterion (BIC) and explained variation statistic R2 , . The AIC is defined in the usual manner as - 2Log(likelihood) + 2(No. of model parameters), while BIC equals - 2Log(likelihood) + (No. of model parameters)*Log(n). In survival analysis n is interpreted as the number of events rather than the number of patients. The placement of knots in spline modeling is an issue. We have placed the knots at equally spaced centiles of the log-survival times, following published recommendations . For example, for m = 1 the knot is at the 50th centile, for m = 2 the knots are at the 33th and 67th centiles, etc.
We compared RP and CPH by performing internal validation (assessing validity in the population where the development data originated from) on the whole data set (naïve) and using split-sample cross-validation. We performed 10-fold cross-validation by splitting the data into development and validation sets and repeating the process 20 times. The methods can be readily implemented in Stata ,  statistical software using the stpm  and stpm2  commands, or in open source statistical software R as flexsurv package .
Assessment of model performance
Model performance is the ability of the estimated risk score to predict survival and is assessed using the measures of explained variation, calibration, and discrimination. Calibration refers to how closely the predicted survival at a pre-specified time agrees with the observed survival. For cross-validation, we compared the average fitted probabilities of survival under RP and CPH for the first 15 days to observed probabilities estimated non-parametrically using Kaplan-Meier curves .
The Brier score is a quadratic scoring rule that calculates the differences between the actual outcomes and predicted probabilities. Given the predicted probability of survival pi at time t for patient i, and Yi binary (0–1, dead-alive) variable, the Brier score is defined as . A Brier score of 0 indicates a perfect model, while 0.25 indicates a non-informative model (the value achieved when issuing a predicted probability of 50% to each patient). The Brier score may be scaled by its maximum Briermax = (1 – mean(pi)) mean(pi) to obtain . The scaled Brier scores range from 0% to 100% and have interpretation similar to the Pearson correlation coefficient.
For a particular risk score, discrimination is the ability to differentiate between the patients who died versus those who survived. The Kaplan-Meier plot of survival for patients in different risk groups can be used to test for separation, indicating that the different risk groups are well defined . For a statistical model, the global measure of the model's discriminatory power is the explained variation statistic R2, which measures the variation explained by the fitted model . Higher values of R2 indicate greater discrimination. In this study we implement R2 for survival models, as described by Royston and Sauerbrei,.
The discrimination or Yates slope is a measure of how well the subjects with and without the outcome are separated. It is defined as the absolute difference in mean predictions of survival (mean[pi]) between those who died and those who survived at time t. The scaled Brier scores and discrimination slopes were calculated separately for the (naïve) model using the whole dataset and the model derived using cross-validation for t = 1, 2… 100 days. Higher scaled Brier scores and discrimination slopes represent better model performance.
Description of the data source
The patient characteristics of the retrospective cohort are summarized in Table 1. The cohort consisted of 293 males (49.7%) and 295 females (50.0%), and 2 (0.3%) with unknown gender. The data were collected starting from patients' entry into hospice care until death for all 590 patients. The mean, median and range of survival times for the patients by PPS at admission, age, gender, cancer status, and diagnosis category are given in Table 2. The table shows that the median survival was fairly evenly distributed across age groups and gender, but unevenly across the cancer status and initial diagnosis category. All patients were assigned PPS at the time of admission to hospice care. Since PPS score of 0% means that the patient is dead, the data were transformed so that the PPS score of 10% was set as the baseline. There were only 15 total observations for PPS = 60%, 70%, 80%, so they were combined with PPS = 50% to obtain meaningful survival estimates. Fourteen patients had missing values for PPS.
Table 1. Patient characteristics.doi:10.1371/journal.pone.0047804.t001
Table 2. Survival time by age, gender, diagnosis and initial PPS.doi:10.1371/journal.pone.0047804.t002
The time of admission was the starting point for survival time. The Kaplan-Meier curves stratified by initial PPS level are shown in Figure 1. The curves show good separation indicating that the different risk groups are well defined. The log-rank test for equality of survival curves was highly significant at P = 0.001. The global test based on Schoenfeld residuals showed that the proportional hazard assumption was violated for PPS (P-value <0.001), which can also be seen from the un-parallel natural log-plot of survival curves (Figure 2).
Figure 1. Kaplan-Meier survival curves by initial PPS.doi:10.1371/journal.pone.0047804.g001
Figure 2. Test of the proportional hazards assumption under CPH for initial PPS.doi:10.1371/journal.pone.0047804.g002
Table 3 lists AIC, BIC and R2 values for 5 knots under the proportional hazard, proportional odds and probit RP families; the minimum combination in each is underlined. The number of optimal knots was found to be m = 1 under the probit model. The improvement in fit with the probit model can be seen from the parallel survival curves of log-probit against natural log time (Figure 3).
Figure 3. Test of the probit assumption under RP for initial PPS.doi:10.1371/journal.pone.0047804.g003
Table 3. Criteria for the choice of scale in the RP model.doi:10.1371/journal.pone.0047804.t003
R2 was higher in the RP model (R2 = 0.298; 95% CI: 0.236–0.358) than the Cox model (R2 = 0.156; 95% CI: 0.111–0.203), indicating that the RP model explained significantly more variation than CPH. To illustrate the differences for the baseline function, Figure 4 shows plots of the CPH and RP baseline survival functions. The CPH baseline survival is “choppy” to approximately day 12, while the RP is smooth. The two baseline functions converged at around day 12.
Figure 4. Baseline survival functions under CPH and RP models.doi:10.1371/journal.pone.0047804.g004
Cross-validation showed that the relation between the two predicted survival estimates is approximately linear, with RP model consistently estimating a higher probability, which is particularly evident for higher scores of PPS corresponding to longer survival times (Figure 5). Overall, the predicted probabilities under RP tended to be closer to the Kaplan-Meier estimates than CPH. The plot of the consistently positive differences between RP and CPH scaled Brier scores (Figure 6a) and discrimination slopes (Figure 6b) showed that the RP model discriminated better across patient survival times for both the full (naïve) and cross-validated models. This suggested that the higher value of R2 under RP was not due to over-fitting.
Figure 5. Predicted survival by PPS under RP and CPH compared with the Kaplan-Meier estimates in the validation data.doi:10.1371/journal.pone.0047804.g005
Figure 6. Difference between Brier scores for RP and CPH models (6a) and between discrimination slopes for RP and CPH models (6b) as a function of patient survival times in the naïve (whole) data set and cross-validated data set.
Both are consistently higher for RP indicating better accuracy and discrimination.doi:10.1371/journal.pone.0047804.g006
The results from our study show that RP family of models predicts survival more accurately than CPH through its flexible modeling of the baseline survival function. Using the RP flexible baseline function modeling would allow for more precise calibration in the prognostication phase than CPH. As Figure 5 illustrates, the predicted RP survival probabilities are consistently higher for higher values of PPS, and closer to the Kaplan-Meier estimates of survival. We suspect that both the robust modeling of baseline survival and overall model fit provide for better survival estimation.
There are limitations to our study, the primary one being the use of retrospective data. The RP family of parametric functions needs to be applied prospectively to assess accuracy of prognostic models through external validation. Furthermore, the dataset was limited to the hospice setting with no censored observations and with majority of patients having a very short follow-up time. For future studies, application of the proposed methodology should account for these limitations, and comparisons with parametric prognostic survival models should be explored.
The flexible models discussed in this paper could greatly improve the ability of researchers to accurately predict survival. An advantage of RP is that it can be used to validate published models for which the original individual patient data are unavailable. If the scale used (hazard, probit or odds), the knot positions, and the estimates of prognostic indices are known, then it would be possible to use RP. In the case of CPH this is not possible, since the baseline function would not be available.
Conceived and designed the experiments: BM BD. Analyzed the data: BM. Contributed reagents/materials/analysis tools: RS SK. Wrote the paper: BM BD AK RM.
- 1. Royston P, Moons KG, Altman DG, Vergouwe Y (2009) Prognosis and prognostic research: Developing a prognostic model. BMJ 338: b604. doi: 10.1136/bmj.b604
- 2. Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, et al. (2010) Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology 21: 128–138. doi: 10.1097/ede.0b013e3181c30fb2
- 3. Vickers AJ (2011) Prediction models: revolutionary in principle, but do they do more good than harm? J Clin Oncol 29: 2951–2952. doi: 10.1200/jco.2011.36.1329
- 4. Moons KG, Altman DG, Vergouwe Y, Royston P (2009) Prognosis and prognostic research: application and impact of prognostic models in clinical practice. BMJ 338: b606. doi: 10.1136/bmj.b606
- 5. Steinhauser KE, Christakis NA, Clipp EC, McNeilly M, McIntyre L, et al. (2000) Factors considered important at the end of life by patients, family, physicians, and other care providers. JAMA 284: 2476–2482. doi: 10.1001/jama.284.19.2476
- 6. Christakis NA, Lamont EB (2000) Extent and determinants of error in doctors' prognoses in terminally ill patients: prospective cohort study. BMJ 320: 469–472. doi: 10.1136/bmj.320.7233.469
- 7. Chow E, Harth T, Hruby G, Finkelstein J, Wu J, et al. (2001) How accurate are physicians' clinical predictions of survival and the available prognostic tools in estimating survival times in terminally ill cancer patients? A systematic review. Clin Oncol (R Coll Radiol) 13: 209–218. doi: 10.1007/s001740170078
- 8. Cox DR, Oakes D (1984) Analysis of survival data. London;New York: Chapman and Hall. viii: , 201 p.p.
- 9. Mallett S, Royston P, Waters R, Dutton S, Altman DG (2010) Reporting performance of prognostic models in cancer: a review. BMC Med 8: 21. doi: 10.1186/1741-7015-8-21
- 10. Downing M, Lau F, Lesperance M, Karlson N, Shaw J et al. (2007) Meta-analysis of survival prediction with Palliative Performance Scale. J Palliat Care 23: 245–252; discussion 252–244.
- 11. Lau F, Cloutier-Fisher D, Kuziemsky C, Black F, Downing M, et al. (2007) A systematic review of prognostic tools for estimating survival time in palliative care. J Palliat Care 23: 93–112.
- 12. Royston P, Parmar MK (2002) Flexible parametric proportional-hazards and proportional-odds models for censored survival data, with application to prognostic modelling and estimation of treatment effects. Stat Med 21: 2175–2197. doi: 10.1002/sim.1203
- 13. Anderson F, Downing GM, Hill J, Casorso L, Lerch N (1996) Palliative performance scale (PPS): a new tool. J Palliat Care 12: 5–11.
- 14. Lau F, Downing M, Lesperance M, Karlson N, Kuziemsky C, et al. (2009) Using the Palliative Performance Scale to provide meaningful survival estimates. J Pain Symptom Manage 38: 134–144. doi: 10.1016/j.jpainsymman.2008.05.017
- 15. Harrold J, Rickerson E, Carroll JT, McGrath J, Morales K, et al. (2005) Is the palliative performance scale a useful predictor of mortality in a heterogeneous hospice population? J Palliat Med 8: 503–509. doi: 10.1089/jpm.2005.8.503
- 16. Olajide O, Hanson L, Usher BM, Qaqish BF, Schwartz R, et al. (2007) Validation of the palliative performance scale in the acute tertiary care hospital setting. J Palliat Med 10: 111–117. doi: 10.1089/jpm.2006.0125
- 17. Head B, Ritchie CS, Smoot TM (2005) Prognostication in hospice care: can the palliative performance scale help? J Palliat Med 8: 492–502. doi: 10.1089/jpm.2005.8.492
- 18. Fainsinger RL, Demoissac D, Cole J, Mead-Wood K, Lee E (2000) Home versus hospice inpatient care: discharge characteristics of palliative care patients in an acute care hospital. J Palliat Care 16: 29–34.
- 19. Morita T, Tsunoda J, Inoue S, Chihara S (2001) Effects of high dose opioids and sedatives on survival in terminally ill cancer patients. J Pain Symptom Manage 21: 282–289. doi: 10.1016/s0885-3924(01)00258-5
- 20. Virik K, Glare P (2002) Validation of the palliative performance scale for inpatients admitted to a palliative care unit in Sydney, Australia. J Pain Symptom Manage 23: 455–457. doi: 10.1016/s0885-3924(02)00407-4
- 21. Morita T, Tsunoda J, Inoue S, Chihara S (1999) Validity of the palliative performance scale from a survival perspective. J Pain Symptom Manage 18: 2–3.
- 22. Lau F, Bell H, Dean M, Downing M, Lesperance M (2008) Use of the Palliative Performance Scale in survival prediction for terminally ill patients in Western Newfoundland, Canada. J Palliat Care 24: 282–284.
- 23. Lau F, Maida V, Downing M, Lesperance M, Karlson N, et al. (2009) Use of the Palliative Performance Scale (PPS) for end-of-life prognostication in a palliative medicine consultation service. J Pain Symptom Manage 37: 965–972. doi: 10.1016/j.jpainsymman.2008.08.003
- 24. Ho F, Lau F, Downing MG, Lesperance M (2008) A reliability and validity study of the Palliative Performance Scale. BMC Palliat Care 7: 10. doi: 10.1186/1472-684x-7-10
- 25. Altman DG, Royston P (2000) What do we mean by validating a prognostic model? Stat Med 19: 453–473. doi: 10.1002/(sici)1097-0258(20000229)19:4<453::aid-sim350>3.3.co;2-x
- 26. Altman DG, Vergouwe Y, Royston P, Moons KG (2009) Prognosis and prognostic research: validating a prognostic model. BMJ 338: b605. doi: 10.1136/bmj.b605
- 27. Kalbfleisch JD, Prentice RL (2002)The statistical analysis of failure time data. Hoboken, N.J.: J. Wiley.. xiii: , 439 p.p.
- 28. Binquet C, Abrahamowicz M, Mahboubi A, Jooste V, Faivre J, et al. (2008) Empirical study of the dependence of the results of multivariable flexible survival analyses on model selection strategy. Stat Med 27: 6470–6488. doi: 10.1002/sim.3447
- 29. Royston P (2001) Flexible parametric alternatives to the Cox model, and more. Stata Journal 1: 1–28.
- 30. Schwarz G (1978) Estimating Dimension of a Model. Annals of Statistics 6: 461–464. doi: 10.1214/aos/1176344136
- 31. Durrleman S, Simon R (1989) Flexible regression models with cubic splines. Stat Med 8: 551–561. doi: 10.1002/sim.4780080504
- 32. Stata Version 11 [computer program]. 9 ed. College Station, TX: Stata Corporation; 2010.
- 33. Royston P (2011) Flexible parametric survival analysis using stata : beyond the Cox model. College Station, TX: Stata Press.
- 34. Lambert PC, Royston P (2009) Further development of flexible parametric models for survival analysis. The Stata Journal 9: 265–290.
- 35. Jackson C (2012) Flexible parametric survival models.
- 36. Cook NR, Buring JE, Ridker PM (2006) The effect of including C-reactive protein in cardiovascular risk prediction models for women. Ann Intern Med 145: 21–29. doi: 10.7326/0003-4819-145-1-200607040-00128
- 37. Graf E, Schmoor C, Sauerbrei W, Schumacher M (1999) Assessment and comparison of prognostic classification schemes for survival data. Stat Med 18: 2529–2545. doi: 10.1002/(sici)1097-0258(19990915/30)18:17/18<2529::aid-sim274>3.3.co;2-x
- 38. Hu B, Palta M, Shao J (2006) Properties of R(2) statistics for logistic regression. Stat Med 25: 1383–1395. doi: 10.1002/sim.2300
- 39. Altman DG (2009) Prognostic models: a methodological framework and review of models for breast cancer. Cancer Invest 27: 235–243. doi: 10.1080/07357900802572110
- 40. Royston P (2006) Explained variation for survival models. The Stata Journal 6: 1–14.
- 41. Royston P, Sauerbrei W (2004) A new measure of prognostic separation in survival data. Stat Med 23: 723–748. doi: 10.1002/sim.1621