Improving the Evaluation of Prognostic Indices for Survival Outcomes

Abstract

BACKGROUND
In many clinical settings (especially, in cancer studies), prognostic indices are defined in order to stratify patients with respect to the risk of occurrence of a specific outcome, on the basis of a subset of clinical variables. In the practice, based on the class of risk of the patient, clinicians may decide the patient management or may tailor the treatment. Thus, the prognostic groups must correspond to “clinically relevant” differences with respect to the outcome of interest. In the context of survival outcomes, a good prognostic classification should satisfy at least the following properties: (1) the groups must correspond to “well separated” survival curves, (2) the order of the prognostic levels must be retained in all cohort of the same clinical setting, (3) the groups must be reliable in terms of size, (4) the classification should give a good survival prediction. The assessment of prognostic indices is usually performed by using scores (e.g. the Brier score [1], the c-index [2] or the D-index [3]), which only evaluate one or two of these characteristics.

AIM
In order to have a more comprehensive evaluation of a prognostic index, we defined a new measure of prognostic index evaluation for survival outcomes and its performance has been compared to the one of commonly used scores.

METHODS
The Expected SEParation (ESEP) index is a new score, which represents the expected difference between the survival times of any two patients, given that they belong to “consecutive” risk groups (defined by the prognostic index to evaluate). In the common censored data setting, the estimation of the ESEP index relies on the estimation of the restricted mean survival time, which is performed in practice by using an approach based on pseudo-values [4]. This new score has several advantages: 1) it evaluates properties (1-3) of a prognostic index, 2) it is based on the restricted mean survival time, which can be used even in case of non-proportional hazards assumption, 3) thanks to its definition, the value of the ESEP index can be judged by physicians with respect to their clinical goals.
The performance of the ESEP index was evaluated and compared with respect to several measures defined in the field through an extensive simulation study. The simulated data were generated mimicking challenging issues characterizing real-world settings. The different scenarios were defined by varying: the sample size, the percentage of censored data, the patient stratification. By using cross-validation, the several measures were also compared on public real data (such as, the Whitehall I Study).

RESULTS
Overall, in the simulation study, the ESEP index outperformed the other scores by enabling to identify wrong prognostic classifications, even in case of small sample size and/or high percentage of censored data. The same behaviour was also observed when comparing different prognostic classifications on public real data, such as the Whitehall I Study. In particular, differently from the other measures, the ESEP index maintained the property of clearly distinguishing the performances of the prognostic indices even when considering subsamples of the entire dataset (and thus in case of reduced sample size).

CONCLUSIONS
The ESEP index was able to well assess and discriminate prognostic models on several simulated scenarios and on real data, even in challenging scenarios. Moreover, since it evaluates three out of the four properties of a prognostic index, it is sufficient to use it together with a measure of survival prediction (such as the Brier score) to achieve a complete assessment.