Rost, Lauren M

Abstract Models that estimate the probability of an adverse clinical outcome require an operational cutoff to translate continuous estimated probabilities into discrete labels that can trigger clinical action. Although statistical methods identify ‘optimal’ cut-offs, threshold selection ultimately reflects value judgments regarding harm tolerance, resource allocation, and workflow feasibility. We describe a governance-informed approach to selecting a deployment threshold for an asthma exacerbation (AE) prediction model integrated into clinical workflows. Using prevalence-adjusted performance metrics and real-world provider capacity modeling, we evaluated multiple candidate thresholds and quantified downstream workload and missed-event trade-offs. We demonstrate that statistically optimal thresholds may produce operationally infeasible alert volumes or unacceptable miss rates. We propose a structured threshold governance framework integrating statistical performance, clinical utility, stakeholder input, and human oversight safeguards. This case illustrates how threshold decisions should be treated as organizational governance processes rather than purely technical optimizations.

INTRODUCTION: High-quality colonoscopy requires accurate risk stratification for surveillance per the 2020 US Multi-Society Task Force guidelines and adherence to 2024 American College of Gastroenterology/American Society for Gastrointestinal Endoscopy quality benchmarks. Both are operationally challenging in clinical practice. We developed and validated Colon-Pilot, a large language model—powered clinical decision support system using GPT-4o to automate and standardize both functions. METHODS: The system was evaluated in 2 operational modes: (i) a human-in-the-loop clinical decision support validation of surveillance recommendations for 596 colonoscopies, comparing concordance with 2020 US Multi-Society Task Force guidelines against expert consensus, and (ii) an automated administrative audit applying Colon-Pilot to 42,632 colonoscopies across the Mayo Clinic Health System to calculate 2024 American College of Gastroenterology/American Society for Gastrointestinal Endoscopy priority quality indicators. Recommendations were autogenerated unless predefined safety criteria triggered manual review. RESULTS: Colon-Pilot issued recommendations for 522 of 596 cases (87.6%) and flagged 12.4% for manual review. For automated cases, guideline-concordant accuracy was 97.5% (Cohen κ = 0.970) versus 69.7% (κ = 0.781) for original endoscopist recommendations. Discordant artificial intelligence (AI) cases (n = 13) most often recommended longer-than-appropriate intervals (62%). Applied to the enterprise data set, Colon-Pilot calculated performance exceeding 2024 targets: adenoma detection rate 49.8% (≥35%), sessile serrated lesion detection rate 17.7% (≥6%), bowel preparation adequacy 91.8% (≥90%), and cecal intubation rate 97.4% (≥95%). DISCUSSION: Colon-Pilot demonstrated high fidelity in applying surveillance guidelines and automated quality benchmarking, outperforming unassisted endoscopists in guideline adherence. By combining safety protocols with large-scale automated reporting, it offers a scalable solution for improving both efficiency and quality in colorectal cancer prevention.