Gao, Junyi

Background Predictive models employing machine learning algorithms are increasingly being used in clinical decision making, and improperly calibrated models can result in systematic harm. We sought to investigate the impact of class imbalance correction, a commonly applied preprocessing step in machine learning model development, on calibration and modelled clinical decision making in a large real-world context. Methods A histogram boosted gradient classifier was trained on a highly imbalanced national dataset of >1.8 million patients undergoing surgery, to predict the risk of 90-day mortality and complications after surgery. Class imbalance correction strategies including random oversampling, synthetic minority oversampling technique, random under-sampling, and cost-sensitive learning were compared to the natural distribution ('natural'). Models were tested and compared with classification metrics, calibration plots, decision curve analysis, and simulated clinical impact analysis. Results The natural model demonstrated high performance (AUROC 0.94, 95% CI 0.94--0.95 for mortality; 0.84, 95% CI 0.84--0.85 for complications) and calibration (log loss 0.05, 95% CI 0.04--0.05 for mortality; 0.23, 95% CI 0.23--0.24 for complications). Class imbalance mitigation (CSL, ROS, RUS, and SMOTE) did not improve AUROC or AUPRC but increased recall and F1 scores at the expense of precision and accuracy. However, these methods severely compromised model calibration, leading to significant over-prediction of risks (up to a 62.8 % increase) as further evidenced by increased log loss across all mitigation techniques. Decision curve analysis and clinical scenario testing confirmed that the natural model provided the highest net benefit. Conclusion Class imbalance correction methods result in significant miscalibration, leading to possible harm whenused for clinical decision making.