Machine Learning Methods Outperform Logistic Regression-Based Prediction Models in Patients Undergoing Repair of Ruptured Abdominal Aortic Aneurysms
Nathan L. Liang, MD MS, Mohammad H. Eslami, MD MPH.
University of Pittsburgh Medical Center, Pittsburgh, PA, USA.
Objective: Clinical prediction is an important component of evidence-based and personalized medicine. However, current prediction methods rely on fitting existing data to mathematical models that may not reflect the true relationship between outcomes and predictors. We hypothesize that models derived using machine-learning methods (ML) that do not rely on mathematical fitting will provide more accurate predictions than traditional logistic regression-based methods (LR).
Methods: We utilized data from the 2011-2015 American College of Surgeons National Quality Improvement Project (NSQIP) ruptured abdominal aortic aneurysm (RAAA) database. The primary outcome was 30-day mortality. For the ML method we chose the random forest approach, while our baseline prediction model was fit using traditional LR with a split-sample approach. NSQIP data also contains precalculated mortality probabilities (NSQIP-MP), which we used as an additional comparison to the ML and LR predicted mortalities. The area under the receiver operating curve (AUC) was used as the primary comparison, with a higher AUC suggesting better classification accuracy.
Results: The dataset contained 2167 patients over this time period; 882 symptomatic non-ruptured aneurysms were excluded, resulting in 1285 patients presenting with ruptured aneurysms. Roughly half (N=649, 52%) presented with hypotension and 346 (26.9%) received a preoperative blood transfusion. The median age was 74 (inter-quartile-range 66-81) and the majority were male (N=1008, 78.4%). Overall, 45.9% (N=590) received open rather than endovascular repair although this proportion decreased over time from 50% in 2011 to 35% in 2015. The overall mortality rate was 27.5% (N=354). The provided NSQIP-MP probabilities resulted in an AUC of 0.70. This was similar to our LR-derived prediction model with an AUC of 0.74. The ML model had a higher AUC than either LR or NSQIP-MP at 0.87 (P<0.001, Figure). Predicted out-of-sample misclassification error rates for the LR (28%) and ML models (26%) were similar.
Conclusions: Machine learning methods can outperform traditional regression-based prediction models. These methods have implications for developing both point-of-care based clinical prediction tools as well as patient-specific benchmarks for quality improvement. Further study is needed to determine optimal application of these methods.
Back to 2018 ePosters
