Citations

Abstract Models that estimate the probability of an adverse clinical outcome require an operational cutoff to translate continuous estimated probabilities into discrete labels that can trigger clinical action. Although statistical methods identify ‘optimal’ cut-offs, threshold selection ultimately reflects value judgments regarding harm tolerance, resource allocation, and workflow feasibility. We describe a governance-informed approach to selecting a deployment threshold for an asthma exacerbation (AE) prediction model integrated into clinical workflows. Using prevalence-adjusted performance metrics and real-world provider capacity modeling, we evaluated multiple candidate thresholds and quantified downstream workload and missed-event trade-offs. We demonstrate that statistically optimal thresholds may produce operationally infeasible alert volumes or unacceptable miss rates. We propose a structured threshold governance framework integrating statistical performance, clinical utility, stakeholder input, and human oversight safeguards. This case illustrates how threshold decisions should be treated as organizational governance processes rather than purely technical optimizations.

Objectives/Goals: This study aims to evaluate the performance of a third-party artificial intelligence (AI) product in predicting diagnosis-related groups (DRGs) in a community healthcare system. We highlight a use case illustrating how clinical practice leverages AI-predicted information in unexpected yet advantageous ways and assess the AI predictions accuracy and practical application. Methods/Study Population: DRGs are crucial for hospital reimbursement under the prospective payment model. The Mayo Clinic Health System (MCHS), a network of clinics and hospitals serving a substantial rural population in Minnesota and Wisconsin, has recently adopted an AI algorithm developed by Xsolis (an AI-focused healthcare solution provider). This algorithm, a 1D convolutional neural network, predicts DRGs based on clinical documentation. To assess the accuracy of AI-generated DRG predictions for inpatient discharges, we analyzed data from 930 patients hospitalized at MCHS Mankato between March 2 and May 13, 2024. The Xsolis platform provided the top three DRG predictions for the first 48 hours of each inpatient stay. The accuracy of these predictions was then compared against the final billed DRG codes from the hospital’s records. Results/Anticipated Results: In our validation set, Xsolis achieved a top-3 DRG prediction accuracy of 71% at 24 hours and 81% at 48 hours, which is lower than the originally reported accuracy of 81.1% and 83.3%, respectively. Interestingly, discussions with clinical practice leaders revealed that the most valuable information derived from the AI predictions was the expected geometric mean length of stay (GMLOS), which Xsolis was perceived to predict accurately. In the Medicare system, each DRG is associated with an expected GMLOS, a critical factor for efficient hospital flow planning. A subsequent analysis comparing predicted GMLOS with the actual length of stay showed variances of -0.10 days on day 1 and 0.14 days on day 2, indicating a high degree of accuracy and aligning with clinical practice perceptions. Discussion/Significance of Impact: Our research underscores that clinical practice can leverage AI predictions in unexpected yet beneficial ways. While initially focused on DRG prediction, the associated GMLOS emerged as more significant. This suggests that AI algorithm validation should be tailored to specific clinical needs rather than relying solely on generalized benchmarks.