Citations

Abstract Objectives Traditional laboratory regulatory frameworks provide robust guidance for conventional clinical testing but lack requirements for data documentation for artificial intelligence and machine learning (AI/ML) solutions. This gap creates significant risks since AI solutions directly inherit patterns, biases, and limitations from their training data. Here, we describe the development and implementation of a comprehensive data documentation checklist with accompanying data lineage for use within an AI lifecycle framework for clinical laboratories. Methods Building on the previously established Clinical AI Readiness Evaluator (CARE) framework, we developed a comprehensive data checklist and data lineage methodology through a multiphase process, including (1) comprehensive review of existing AI/ML data documentation frameworks, (2) focused meetings with 3 institutional AI operations teams, and (3) 3 rounds of iterative refinement by our multidisciplinary team. The checklist’s effectiveness was then assessed using 3 diverse AI/ML projects moving through the CARE framework. Results The CARE Data Checklist and Data Lineage provide a structured approach to documenting critical aspects of datasets used in AI/ML projects, including data composition, demographics, collection methods, labeling processes, usage constraints, maintenance requirements, and a data readiness assessment. The checklist addresses unique data-centric challenges of AI/ML applications, facilitating transparency, reproducibility, and regulatory compliance. Conclusions The CARE Data Checklist and Data Lineage serve as both a technical guide and a communication tool bridging gaps between technical and clinical stakeholders. By working on these documents early in the AI lifecycle, laboratories can anticipate and address data-related challenges, ultimately saving time, optimizing resources, and improving the reliability of AI-augmented laboratory solutions.

Abstract Recent work on double descent has challenged classical bias–variance tradeoffs, showing that test error can decrease, increase sharply near the interpolation threshold, and then decrease again as model capacity grows. This phenomenon has been documented in regression and classification, but its relevance to survival analysis remains unclear. Survival data are subject to censoring, which obscures true event times, and widely used models such as the Cox proportional hazards model are optimized via partial likelihoods that emphasize ranking rather than calibrated risk estimation. It is therefore unknown whether double descent occurs in this setting, how censoring influences its manifestation, or how it interacts with standard performance metrics.We investigate these questions using synthetic survival data generated from Weibull hazards with controlled censoring, allowing systematic variation of model capacity from under to over parameterized regimes. While we verified double descent occurs in survival models, calibration plateaus and decouples from discrimination, even under strong $\ell_2$ regularization. This decoupling arises because the Cox partial likelihood optimizes rankings rather than magnitudes, producing extreme risk scores that break the Breslow estimator used to estimate survival probabilities. These results highlight limitations of discrimination-based model selection in survival analysis and underscore the need for calibration-aware evaluation in high-capacity prognostic models.

Abstract Background: While synoptic reports have standardized breast cancer data, a wealth of critical information remains locked in unstructured free-text, including nuanced biomarker details and procedural notes. This data is not easily computable, creating a significant barrier to understanding comprehensive population-level statistics and subtle clinical trends. Traditional data extraction methods are often insufficient to capture this complexity. We developed and validated a novel, flexible AI-powered system using an agent-based architecture to extract, normalize, and discretize this valuable unstructured data, enabling previously infeasible population-scale analytics. Methods: Our system employs a modular, agent-based framework where each "agent" is a specialized large language model (LLM) tasked with a specific function in a sequential workflow. The process begins with an initiator agent that parses the synoptic report. Subsequent agents then perform targeted tasks, such as identifying and normalizing tumor characteristics, procedural details, and biomarker information from free-text fields. A final agent aggregates this information into a structured, computable format. To rigorously test our approach, we first developed a synthetic data generator to create realistic, anonymized report snippets with known ground-truth values. We then evaluated the performance of several leading LLMs of various sizes (e.g., gemini, llama3.3, phi4, deepseek-r1, gemma3) on this synthetic dataset. Results: Our early analysis on simulated synoptic reports demonstrated high accuracy in data extraction and normalization. The gemini-1.5-pro-002 model achieved an overall accuracy of 96.9%, with a sensitivity of 99.3%, specificity of 94.3%, and an F1-score of 97.0%. The llama3.3:70b model also showed strong performance with a 98.7% accuracy, (F1: 98.8%, sensitivity: 99.8%, specifically: 97.6%). Gemma3:27b was the least performant model tested (F1: 66.2%), substantially worse than much smaller models (deepseek-r1:1.5b and phi4:14b). A detailed error analysis of false positives and negatives has allowed us to iteratively refine the agent logic. By the time of the symposium, we will have analyzed a large cohort of real, de-identified synoptic reports and will present comprehensive performance metrics and initial findings on population-level trends. Conclusion: This AI-powered, agent-based system offers a powerful and scalable solution to overcome the limitations of structured data in synoptic reports. By accurately extracting and normalizing previously inaccessible free-text information, our approach has the potential to provide clinicians and researchers with near real-time insights into population-level statistics and trends in breast cancer. This will facilitate a more profound understanding of disease patterns and ultimately support evidence-based advancements in patient care and clinical research. Citation Format: S. N. Hart. A Novel Agent-Based AI System to Unlock Unstructured Data in Synoptic Reports for Advanced Population Health Analysis in Breast Cancer [abstract]. In: Proceedings of the San Antonio Breast Cancer Symposium 2025; 2025 Dec 9-12; San Antonio, TX. Philadelphia (PA): AACR; Clin Cancer Res 2026;32(4 Suppl):Abstract nr PS3-06-11.

Abstract Objectives To develop and validate an agent-based Large Language Model (LLM) system for extracting structured data from breast cancer synoptic pathology reports and assess the performance gap between synthetic and real-world validation. Materials and Methods We developed a modular artificial intelligence (AI) agent-based framework employing sequential specialized LLMs for parsing pathology reports and extracting structured data. We normalized College of American Pathologists (CAP) cancer protocols into 8 sections, 86 subsections, and 229 discrete fields. Seven leading LLMs (gemini-2.5-pro, llama3.3-70b, phi4-14b, deepseek-r1 14B/70B, gemma3-27b, gemini-2.0-flash-lite) were validated using dual evaluation: synthetic validation (864 controlled test cases) and real-world ground truth (6651 annotated fields from 90 pathology reports). Results Synthetic validation demonstrated strong performance (accuracy: 93.8%-99.0%). Real-world evaluation revealed field extraction recall ranging from 61.8% to 87.7%, demonstrating a substantial “reality gap” with performance drops of 11-32 percentage points. The gemini-2.5-pro model achieved the highest real-world recall (87.7%). Model size did not predict performance: the 14B-parameter deepseek-r1 (77.6%) outperformed its 70B-parameter counterpart (70.4%). Discussion The substantial performance degradation from synthetic to real-world data underscores the complexity of authentic clinical documentation. Smaller models can achieve competitive or superior recall, reducing computational costs. With even the best models missing 12%-38% of annotated fields, mandatory human verification is essential for clinical deployment. Conclusion While LLM-based extraction systems show promise for pathology data extraction, synthetic validation alone provides false confidence. Rigorous real-world ground truth evaluation with expert annotation is essential before clinical deployment. These systems are best positioned as screening tools with mandatory human oversight rather than autonomous decision-making systems.

Abstract Purpose: Germline pathogenic variants (PV) in ATM increase the risk of pancreatic ductal adenocarcinoma (PDAC), but the underlying tumor biology of PDAC associated with germline PV in ATM has not been adequately explored. Experimental Design: Whole-genome, whole-exome, and RNA sequencing were performed on PDAC tumors from 25 germline ATM PV carriers diagnosed at Mayo Clinic between 2007 and 2017. Somatic and copy-number alterations, mutational signatures, transcriptomic subtypes, and the immune landscape were evaluated. Results: High-quality whole-exome and whole-genome sequencing were obtained from 21 and 15 tumors, respectively. Biallelic inactivation of ATM was observed in 87%, KRAS PV in 90%, CDKN2A homozygous loss in 60%, and TP53 alterations in <10% of these tumors. A predominant clock-like mutational signature was present in all samples. Whole-transcriptome analysis identified that the aberrantly differentiated endocrine exocrine subtype accounted for 18% of PDAC and was consistently associated with >5-year overall survival. In addition, a 28-gene expression–based signature associated with overall survival was identified and further validated in The Cancer Genome Atlas cohort. Immune landscape analysis through CODEX identified enriched CD4 T-helper cell/tumor interactions and reduced B7H3-high cell/tumor interactions in ATM PV carriers compared with noncarriers. Conclusions: The observed absence of TP53 PV and enrichment for CDKN2A alterations in ATM tumors, along with differences in the mutational signatures, transcriptomic subtypes and immune landscape, improve our understanding of the mechanistic pathways involved in PDAC development in germline ATM PV carriers and help identify potential targeted therapeutic strategies.

Abstract Objective Despite exponential growth in artificial intelligence (AI) research for laboratory medicine and pathology, a significant gap exists between model development and clinical AI implementation. This article introduces a structured framework, the Clinical AI Readiness Evaluator (CARE), to bridge this gap and support the responsible adoption of AI in clinical laboratory settings. Methods Building upon the Machine Learning Technology Readiness Levels framework, we developed CARE specifically for the clinical laboratory environment by incorporating health care–specific requirements, regulatory considerations, and workflow integration needs. This framework was iteratively refined through practical application across diverse AI use cases within laboratory medicine and pathology. Results The CARE framework provides a systematic approach to AI development and implementation through 8 component workstreams: clinical use case, data, data pipeline, code, clinical user experience, clinical technology infrastructure, clinical orchestration, and regulatory compliance. Unlike generic AI frameworks, CARE distinguishes itself by emphasizing both health care and laboratory workflow integration, regulatory requirements, ethical considerations, and comprehensive validation for clinical contexts. The framework accommodates both internally developed models and commercial AI solutions, providing clear guidance through technology readiness levels and structured review processes. Conclusions The CARE framework addresses the unique challenges of implementing AI in laboratory medicine and pathology by providing a comprehensive roadmap from initial concepts through clinical deployment and maintenance. This article, the first in a series of 4, establishes the foundational AI lifecycle framework, while subsequent articles will explore data documentation, ethical AI considerations, and governance structures. By adopting this structured approach, laboratories can responsibly harness AI’s potential to enhance diagnostic accuracy and operational efficiencies and, ultimately, improve patient care.

Abstract Background The loss-of-function (LOF) classification of most missense variants in tumor suppressor breast cancer genes BRCA1, BRCA2, PALB2 , and RAD51C remains unclassified and confounds clinical actionability. Classifying these variants is challenging due to their rarity, leading clinicians to rely on in silico predictive methods. Protein stability changes are associated with function, making stability predictors valuable. Stability predictions upon missense variant perturbations require high-resolution protein structures. However, the availability of these high-resolution structures is lacking. This study explores using generative AI to predict high-resolution protein structures, which can then be analyzed with in silico protein stability prediction methods to assess LOF activity in ordered regions of the protein. This study also determines the appropriate in silico protein stability and dedicated in silico missense prediction methods in dbNSFP v4.7 database to predict LOF activity in ordered regions of these four genes. Functional classifications from homology recombination DNA repair (HDR) assays and variant classifications from the ClinVar database provide a reliable dataset for evaluating the performance of these in silico prediction methods. Results Complex AlphaFold2 structures of the BRCA1-C terminal (BRCT) domain and the DNA-binding (DB) domain of BRCA2, analyzed using protein stability tool FoldX predicts LOF activity from missense variants significantly better than experimentally-derived structures in ordered regions. The BRCT domain achieved an Area Under the Curve (AUC)= 0.861 (95 % CI:0.858–0.863) and AUC= 0.842 (95 % CI:0.840–0.845), while the DB domain achieved an AUC= 0.836 (95 % CI:0.8322–0.841), compared to AUC= 0.847 (95 % CI:0.844–0.850) and AUC= 0.835 (95 % CI:0.832–0.837) from the BRCT domain, and AUC= 0.830 (95 % CI:0.821–0.8320) from the DB domain from experimentally-derived structures. Protein stability does not predict LOF activity from missense variants better than dedicated in silico missense predictors. Overall, we find that AlphaMissense ranks highly, with an average AUC= 0.890 (95 % CI 0.886–0.895) from ordered regions across these four cancer genes, compared to all other in silico missense predictors present in the dbNSFP database. Conclusions The study reveals that generative AI protein predicted structures can outperform experimentally-derived structures in evaluating LOF activity from predicted protein stability in ordered regions of genes BRCA1, BRCA2, PALB2 and RAD51C. The study also highlights the predictive performance of AlphaMissense as the premier in silico missense prediction method to predict LOF activity from missense variants in these four tumor suppressor breast cancer genes. The code for this study can be downloaded for free on GitHub ( https://github.com/rohandavidg/CarePred )

Context.— Generative artificial intelligence (AI) technologies are rapidly transforming numerous fields, including pathology, and hold significant potential to revolutionize educational approaches. Objective.— To explore the application of generative AI, particularly large language models and multimodal tools, for enhancing pathology education. We describe their potential to create personalized learning experiences, streamline content development, expand access to educational resources, and support both learners and educators throughout the training and practice continuum. Data Sources.— We draw on insights from existing literature on AI in education and the collective expertise of the coauthors within this rapidly evolving field. Case studies highlight practical applications of large language models, demonstrating both the potential benefits and unique challenges associated with implementing these technologies in pathology education. Conclusions.— Generative AI presents a powerful tool kit for enriching pathology education, offering opportunities for greater engagement, accessibility, and personalization. Careful consideration of ethical implications, potential risks, and appropriate mitigation strategies is essential for the responsible and effective integration of these technologies. Future success lies in fostering collaborative development between AI experts and medical educators, prioritizing ongoing human oversight and transparency to ensure that generative AI augments, rather than supplants, the vital role of educators in pathology training and practice.

Abstract Breast cancer is one of the most common cancers in women. With early diagnosis, some breast cancers are highly curable. Classifying normal versus tumor breast tissues from microscopy images is an ideal case to use for deep learning and could help diagnose breast cancer with higher reproducibility. Since data preprocessing and hyperparameter tuning have impacts on breast cancer classification accuracies, training a classifier with appropriate data preprocessing and optimized hyperparameters could improve breast cancer classification accuracy. Using 12 combinations of model architectures, data preprocessing, and hyperparameter configurations, the validation accuracy was calculated using the BreAst Cancer Histology (BACH) dataset. The DenseNet201, a non-specialized model architecture, with transfer learning approach achieved 98.61% validation accuracy compared to only 64.00% for the digital pathology (DP)-specialized model architecture. The combination of image data preprocessing and hyperparameters have a profound impact on the performance of deep neural networks for image classification. To identify a well-performing neural network to classify tumor versus normal breast histology, researchers should not only focus on developing new models specifically for DP, since hyperparameter tuning for existing deep neural networks in the computer vision field could also achieve a better prediction accuracy.

Context.— Computational pathology combines clinical pathology with computational analysis, aiming to enhance diagnostic capabilities and improve clinical productivity. However, communication barriers between pathologists and developers often hinder the full realization of this potential. Objective.— To propose a standardized framework that improves mutual understanding of clinical objectives and computational methodologies. The goal is to enhance the development and application of computer-aided diagnostic (CAD) tools. Design.— This article suggests pivotal roles for pathologists and computer scientists in the CAD development process. It calls for increased understanding of computational terminologies, processes, and limitations among pathologists. Similarly, it argues that computer scientists should better comprehend the true use cases of the developed algorithms to avoid clinically meaningless metrics. Results.— CAD tools improve pathology practice significantly. Some tools have even received US Food and Drug Administration approval. However, improved understanding of machine learning models among pathologists is essential to prevent misuse and misinterpretation. There is also a need for a more accurate representation of the algorithms’ performance compared to that of pathologists. Conclusions.— A comprehensive understanding of computational and clinical paradigms is crucial for overcoming the translational gap in computational pathology. This mutual comprehension will improve patient care through more accurate and efficient disease diagnosis.