A note on prognostic accuracy evaluation of regression models applied to longitudinal autocorrelated binary data

Abstract

Background: Focus of this work was on evaluating the prognostic accuracy of two approaches for modelling binary longitudinal outcomes, a Generalized Estimating Equation (GEE) and a likelihood based method, Marginalized Transition Model (MTM), in which a transition model is combined with a marginal generalized linear model describing the average response as a function of measured predictors.

Methods: A retrospective study on cardiovascular patients and a prospective study on sciatic pain were used to evaluate discrimination by computing the Area Under the Receiver-Operating-Characteristics curve, (AUC), the Integrated Discrimination Improvement (IDI) and the Net Reclassification Improvement (NRI) at different time occasions. Calibration was also evaluated. A simulation study was run in order to compare model’s performance in a context of a perfect knowledge of the data generating mechanism.

Results: Similar regression coefficients estimates and comparable calibration were obtained; an higher discrimination level for MTM was observed. No significant differences in calibration and MSE (Mean Square Error) emerged in the simulation study, that instead confirmed the MTM higher discrimination level.

Conclusions: The choice of the regression approach should depend on the scientific question being addressed, i.e. if the overall population-average and calibration or the subject-specific patterns and discrimination are the objectives of interest, and some recently proposed discrimination indices are useful in evaluating predictive accuracy also in a context of longitudinal studies