Improving Machine Learning and Deep Learning Models for 30-Day ICU Readmission Prediction Using Ensemble Bayesian Model Averaging

Abstract

Introduction

Intensive Care Unit (ICU) readmissions increase morbidity, mortality, and healthcare costs. Existing deep learning (DL) models for predicting 30-day ICU readmission show limited clinical utility due to poor calibration and overconfident predictions that can mislead clinicians. While combining multiple predictive models to improve overall performance is a well-established and often effective strategy, the most widely used ensemble approaches overlook model uncertainty, which can still lead to overconfident decision-making. In contrast, Ensemble Bayesian Model Averaging (EBMA) provides a robust framework for incorporating model uncertainty. While EBMA rewards discriminative power, it also penalizes models that provide overconfident yet inaccurate predictions, resulting in more reliable and clinically meaningful risk estimates.

Aims

To develop an EBMA-based framework to integrate predictions of 30-day ICU readmission from multiple DL, machine learning (ML), and traditional scoring models.

Methods

We analyzed first ICU admissions for adult patients (≥18 years) from the MIMIC-III database (n=32,784 stays). We randomly split the data into training (80%), calibration (13%), and test (7%) sets. Drawing from our own and other systematic reviews, we adapted 17 DL models — including RNN-based architectures, CNN-LSTM combinations, and a transformer-based model — and 6 ML models (two logistic regression models, Naïve Bayes, random forest, support vector machine, and XGBoost), training all models on the same training set. Their predictions, together with the traditional SWIFT and APACHE-II scores, served as input for our EBMA-based framework. Model weights were then estimated on the calibration set. We assessed the calibration and discrimination of the obtained ensemble on the test set using the Brier Score and AUROC, respectively. We shared the code supporting this study at https://github.com/ekoumantakis/ebma-icu-readmission.

Results

Our EBMA-based framework achieved a Brier Score of 0.051 and an AUROC of 0.728, outperforming all individual models in calibration while matching or exceeding their discrimination performance. These results remained robust after excluding lower-weighted models and, notably, even after removing the highest-weighted and best-calibrated model. Compared with individual models, similarly well-calibrated models (i.e., with similar Brier Scores) achieved AUROC values of at most 0.683, indicating inferior discrimination. In contrast, the most discriminative model (AUROC = 0.716) exhibited poor calibration (Brier Score 0.148). While our EBMA-based framework achieved only a marginal improvement in AUROC over this model (+1.7%), it substantially improved calibration, yielding more reliable and clinically valuable predictions.

Conclusions

Well-calibrated predictions are more valuable for clinical decision-making than high discrimination alone. Our EBMA-based framework reliably integrates multiple models, delivering superior calibration while maintaining a discrimination comparable to the best-performing individual model. Our framework shifts away from the “winner-takes-all” paradigm of standalone models toward a more robust, ensemble-based strategy. Future work should focus on external validation and on boosting model diversity to further improve discriminative performance.