Risk Scores Validation: An Example in Cardiology

Abstract

Introduction: Clinicians often make important decisions about patient care by estimating the likelihood of a particular disease, condition or event occurring. Prediction models are useful in this context. Development studies aim to develop a prediction model by selecting clinically relevant predictors and statistically combining them in a multivariable logistic or cox model [1]. Once a model has been developed, its performance must be assessed in the same cohort (internal validation) and in a new cohort (external validation). The performance of a model can be assessed in terms of calibration (comparison between the observed and predicted proportions of events) and discrimination (ability to predict patients who will or will not have the event of interest) [2]. There are several calibration methods: calibration in large, calibration curve associated with the Hosmer-Lemeshow test and calibration slope. Discrimination can be assessed by the area under the ROC curve (AUC) or the Harrel-c index, depending on the regression model used for development [3]. The development and validation of predictive scores are useful in cardiology: a study was conducted to investigate whether the ECG acquired after return of spontaneous circulation (ROSC) could play a prognostic role for 30-days mortality in patients surviving from out-of-hospital cardiac arrest (OHCA), defined as sudden cessation of cardiac function with loss of consciousness and circulatory signs occurring in an out-of-hospital setting. The study was conducted considering post-ROSC ECGs related to OHCA patients from 2015 to 2018 in the populations of Lugano, Vienna and Pavia. Multivariable cox regression was performed and age ≥62 years, female, ECG acquisition time≥8 min, presence of >1 segment with ST elevation, a QRS≥120 msec and the diagnostic pattern for Brugada syndrome resulted associated with higher 30-days mortality. The coefficient of each variable was multiplied by 10 and rounded to the nearest whole number and, by summing the rounded coefficients, a score between 0 and 26 was created. The study showed that the risk of death increased as the score increased. Furthermore, by dividing the population according to score tertiles, 30-days mortality risk classes were identified: low (score 0-4), intermediate (score 5-7) and high (score 8-26) [4].

Aim: The aim of this study is to validate a post-ROSC ECG score in predicting mortality risk and stratifying 30-days mortality risk after OHCA in a new cohort.

Methods: This is a multicenter, prospective, score validation study to predict 30-days mortality in OHCA survivors. Post-ROSC ECGs of patients enrolled in the LombardiaCARe registry from 01/01/2015 to 31/12/2023 and ECGs of OHCA patients admitted to Saint-Pierre Hospital, Brussels, from 01/01/2017 to 31/12/2023 were collected. The same outcome and the same predictors of the previous work were considered. Categorical variables were described as numbers and percentages and compared using the chi-squared test or the Fisher exact test, depending on the expected frequencies. Continuous variables were described as mean ± standard deviation and compared with the t-test or described as median and interquartile range (IQR) and compared with the Mann-Whitney test and according to their normal distribution, tested with the Shapiro-Wilk test. The risk score and mortality risk groups were identified according to previous work [4]. Univariable cox regression was performed with 30-days mortality risk category as the independent variable. The assumption of hazard proportionality was tested using the Shoenfeld test. Calibration was assessed by plotting the observed proportions of events against the predicted probabilities, while the c-index was assessed for discrimination. Moreover, the prognostic index (PI) was calculated from the cox regression model with the same predictors of the previous work [4] as covariates and Kaplan–Meier (KM) curves of PI tertiles were plotted. Long-rank test was used to test the difference between the three curves. All values p<0.05 were considered statistically significant. Statistical analyses were performed using Stata17.

Results:  A total of 1167 ECGs were collected in the two centres and score calculation was possible for 1075 of them. Of these patients, 431 (40.1%) were alive at 30 days. The median score was 10.0 (6.0-12.0) and 175 (16.35%) patients were classified as low risk, 300 (27.9%) as intermediate risk and 600 (55.8%) as high risk. Cox regression showed that patients in the intermediate risk group had a higher risk of death compared with those in the low risk group (HR: 1.3 [95% CI: 1.1-1.9]; p-value: 0.049), as did patients in the high risk group (HR: 1.9 [95% CI: 1.4-2.5]; p-value<0.001). The harrel-C of the model is C:0.56 [95% CI: 0.54-0.59].

Conclusion: The discrimination is lower compared to the original model (Harrel-c: 0.66 [95% CI, 0.57-0.76]), though acceptable. Indeed, the model retains the ability to discriminate patients at low, intermediate and high risk of 30-days mortality. Figure 1A shows the KM curves of the PI tertiles (p-value<0.001) and the graph suggests that the model discriminates better patients at hight risk rather than low and intermediate risk, as expected considering the harrel-c. Figure 1B shows similar predicted and observed survival probabilities in all groups, confirming good calibration of the model. A limitation of the study is the non-homogeneous distribution of patients in the 3 risk groups. Our results suggest that the post-ROSC ECG score can predict the risk of 30-days mortality after OHCA. This provides a possibility for risk stratification in post-cardiac arrest care, assisting clinicians in clinical decision making and underlining the prognostic role of ECG.