Prediction of Suspected Choledocholithiasis: A Comparison of Diagnostic Performance of ML Algorithms Against Current Guidelines

Abstract

Introduction Choledocholithiasis or common bile duct stones (CBDS) is a frequent cause of hospitalization. Confirmed stones are usually removed by Endoscopic Retrograde Cholangiopancreatography (ERCP), but this procedure carries significant risks (infection, perforation, hemorrhage). Current guidelines from the European and American Societies for Gastrointestinal Endoscopy (ESGE and ASGE) suggest ERCP only in patients deemed at high-likelihood of CBDS. 2019 updated ESGE guidelines define high-likelihood as having cholangitis or suspected CBDS at ultrasound (US), while ASGE also requires the presence of both total bilirubin >4mg/dL and CBD dilation on US. However, these criteria do not show conclusive performance with all studies showing very low sensitivity and high false negative rate although showing high specificity (thus minimizing the risk of unnecessary ERCPs).

Objective The objective of the present study was twofold: 1) to evaluate the diagnostic performance of ASGE and ESGE 2019 criteria in confirming suspected CBDS on a novel single-center retrospective cohort; 2) to train different machine learning (ML) models and evaluate their performances in comparison with traditional guidelines.

Methods Data extracted from the Morgagni-Pierantoni Hospital database, included n=870 patients admitted between 2017 and 2022, with 13% later confirmed CBDS. The study included patients who underwent US and blood tests and had a minimum follow-up of 12 months. Variables used for ML training were those that define ESGE/ASGE likelihood plus age, sex, white blood cells (WBC) and C-reactive protein (CRP) and were distributed at baseline as follows: mean age 66±17, female sex 52%, cholangitis 12%, abnormal liver function tests 34%, CBDS on US 4.6%, CBD dilation on US 17%, mean total bilirubin 1.30±1.73mg/dL, mean WBC 10480±6276, mean CRP 76±105. 14% (ESGE) and 15% (ASGE) scored as high risk. Outcome definition was confirmed CBDS at intraoperative RX or ERCP. ML algorithms compared were LR, SVM, KNN, LightGBM, Random Forest, GradientBoosting, XGBoost and MLP. Training, validation and test split was 70%/15%/15%; 10-fold cross validation was used. AUC and Youden Index derived sensitivity, specificity, PPV, NPV and accuracy with 95% confidence intervals (CI) were used for evaluation. 

Results The only model which achieved better results at all diagnostic measures was a deep feed-forward MLP architecture (layers size 32,16) trained on normalized predictor values (batch size=32, AdamW optimizer, cross-entropy loss). It showed good calibration (Brier score=0.14) and reached a test AUC of 0.85 [0.79-0.91]. At the best probability threshold (0.22) it reached better test accuracy (0.85 [0.80-0.88]) than ESGE (0.82 [0.79-0.84]) and ASGE (0.82 [0.80-0.85]). It also showed significantly higher test sensitivity (0.55 [0.39-0.71]) than ESGE (0.35 [0.27-0.44]) and ASGE (0.4 [0.32-0.49]). Similar specificity was observed (MLP 0.89 [0.87-0.92]), ESGE 0.89 [0.87-0.91], ASGE 0.89 [0.86-0.91]. Test PPV and NPV were also significantly better (MLP 0.42 [0.33-0.52] and 0.93 [0.91-0.95]; ESGE 0.32 [0.22-0.44] and 0.90 [0.87-0.91]; ASGE 0.34 [0.24-0.45] and 0.91 [0.89-0.91]).

Conclusions We compared 8 ML models to ASGE and ESGE high-likelihood criteria for suspected CBDS and showed that MLP reached significantly better performance in terms of sensitivity, NPV and PPV and overall greater accuracy while keeping similar specificity.