Hart, Steven N

Abstract Objectives Traditional laboratory regulatory frameworks provide robust guidance for conventional clinical testing but lack requirements for data documentation for artificial intelligence and machine learning (AI/ML) solutions. This gap creates significant risks since AI solutions directly inherit patterns, biases, and limitations from their training data. Here, we describe the development and implementation of a comprehensive data documentation checklist with accompanying data lineage for use within an AI lifecycle framework for clinical laboratories. Methods Building on the previously established Clinical AI Readiness Evaluator (CARE) framework, we developed a comprehensive data checklist and data lineage methodology through a multiphase process, including (1) comprehensive review of existing AI/ML data documentation frameworks, (2) focused meetings with 3 institutional AI operations teams, and (3) 3 rounds of iterative refinement by our multidisciplinary team. The checklist’s effectiveness was then assessed using 3 diverse AI/ML projects moving through the CARE framework. Results The CARE Data Checklist and Data Lineage provide a structured approach to documenting critical aspects of datasets used in AI/ML projects, including data composition, demographics, collection methods, labeling processes, usage constraints, maintenance requirements, and a data readiness assessment. The checklist addresses unique data-centric challenges of AI/ML applications, facilitating transparency, reproducibility, and regulatory compliance. Conclusions The CARE Data Checklist and Data Lineage serve as both a technical guide and a communication tool bridging gaps between technical and clinical stakeholders. By working on these documents early in the AI lifecycle, laboratories can anticipate and address data-related challenges, ultimately saving time, optimizing resources, and improving the reliability of AI-augmented laboratory solutions.

Abstract Recent work on double descent has challenged classical bias–variance tradeoffs, showing that test error can decrease, increase sharply near the interpolation threshold, and then decrease again as model capacity grows. This phenomenon has been documented in regression and classification, but its relevance to survival analysis remains unclear. Survival data are subject to censoring, which obscures true event times, and widely used models such as the Cox proportional hazards model are optimized via partial likelihoods that emphasize ranking rather than calibrated risk estimation. It is therefore unknown whether double descent occurs in this setting, how censoring influences its manifestation, or how it interacts with standard performance metrics.We investigate these questions using synthetic survival data generated from Weibull hazards with controlled censoring, allowing systematic variation of model capacity from under to over parameterized regimes. While we verified double descent occurs in survival models, calibration plateaus and decouples from discrimination, even under strong $\ell_2$ regularization. This decoupling arises because the Cox partial likelihood optimizes rankings rather than magnitudes, producing extreme risk scores that break the Breslow estimator used to estimate survival probabilities. These results highlight limitations of discrimination-based model selection in survival analysis and underscore the need for calibration-aware evaluation in high-capacity prognostic models.

Abstract Background: While synoptic reports have standardized breast cancer data, a wealth of critical information remains locked in unstructured free-text, including nuanced biomarker details and procedural notes. This data is not easily computable, creating a significant barrier to understanding comprehensive population-level statistics and subtle clinical trends. Traditional data extraction methods are often insufficient to capture this complexity. We developed and validated a novel, flexible AI-powered system using an agent-based architecture to extract, normalize, and discretize this valuable unstructured data, enabling previously infeasible population-scale analytics. Methods: Our system employs a modular, agent-based framework where each "agent" is a specialized large language model (LLM) tasked with a specific function in a sequential workflow. The process begins with an initiator agent that parses the synoptic report. Subsequent agents then perform targeted tasks, such as identifying and normalizing tumor characteristics, procedural details, and biomarker information from free-text fields. A final agent aggregates this information into a structured, computable format. To rigorously test our approach, we first developed a synthetic data generator to create realistic, anonymized report snippets with known ground-truth values. We then evaluated the performance of several leading LLMs of various sizes (e.g., gemini, llama3.3, phi4, deepseek-r1, gemma3) on this synthetic dataset. Results: Our early analysis on simulated synoptic reports demonstrated high accuracy in data extraction and normalization. The gemini-1.5-pro-002 model achieved an overall accuracy of 96.9%, with a sensitivity of 99.3%, specificity of 94.3%, and an F1-score of 97.0%. The llama3.3:70b model also showed strong performance with a 98.7% accuracy, (F1: 98.8%, sensitivity: 99.8%, specifically: 97.6%). Gemma3:27b was the least performant model tested (F1: 66.2%), substantially worse than much smaller models (deepseek-r1:1.5b and phi4:14b). A detailed error analysis of false positives and negatives has allowed us to iteratively refine the agent logic. By the time of the symposium, we will have analyzed a large cohort of real, de-identified synoptic reports and will present comprehensive performance metrics and initial findings on population-level trends. Conclusion: This AI-powered, agent-based system offers a powerful and scalable solution to overcome the limitations of structured data in synoptic reports. By accurately extracting and normalizing previously inaccessible free-text information, our approach has the potential to provide clinicians and researchers with near real-time insights into population-level statistics and trends in breast cancer. This will facilitate a more profound understanding of disease patterns and ultimately support evidence-based advancements in patient care and clinical research. Citation Format: S. N. Hart. A Novel Agent-Based AI System to Unlock Unstructured Data in Synoptic Reports for Advanced Population Health Analysis in Breast Cancer [abstract]. In: Proceedings of the San Antonio Breast Cancer Symposium 2025; 2025 Dec 9-12; San Antonio, TX. Philadelphia (PA): AACR; Clin Cancer Res 2026;32(4 Suppl):Abstract nr PS3-06-11.

Abstract Objectives To develop and validate an agent-based Large Language Model (LLM) system for extracting structured data from breast cancer synoptic pathology reports and assess the performance gap between synthetic and real-world validation. Materials and Methods We developed a modular artificial intelligence (AI) agent-based framework employing sequential specialized LLMs for parsing pathology reports and extracting structured data. We normalized College of American Pathologists (CAP) cancer protocols into 8 sections, 86 subsections, and 229 discrete fields. Seven leading LLMs (gemini-2.5-pro, llama3.3-70b, phi4-14b, deepseek-r1 14B/70B, gemma3-27b, gemini-2.0-flash-lite) were validated using dual evaluation: synthetic validation (864 controlled test cases) and real-world ground truth (6651 annotated fields from 90 pathology reports). Results Synthetic validation demonstrated strong performance (accuracy: 93.8%-99.0%). Real-world evaluation revealed field extraction recall ranging from 61.8% to 87.7%, demonstrating a substantial “reality gap” with performance drops of 11-32 percentage points. The gemini-2.5-pro model achieved the highest real-world recall (87.7%). Model size did not predict performance: the 14B-parameter deepseek-r1 (77.6%) outperformed its 70B-parameter counterpart (70.4%). Discussion The substantial performance degradation from synthetic to real-world data underscores the complexity of authentic clinical documentation. Smaller models can achieve competitive or superior recall, reducing computational costs. With even the best models missing 12%-38% of annotated fields, mandatory human verification is essential for clinical deployment. Conclusion While LLM-based extraction systems show promise for pathology data extraction, synthetic validation alone provides false confidence. Rigorous real-world ground truth evaluation with expert annotation is essential before clinical deployment. These systems are best positioned as screening tools with mandatory human oversight rather than autonomous decision-making systems.

Abstract Purpose: Germline pathogenic variants (PV) in ATM increase the risk of pancreatic ductal adenocarcinoma (PDAC), but the underlying tumor biology of PDAC associated with germline PV in ATM has not been adequately explored. Experimental Design: Whole-genome, whole-exome, and RNA sequencing were performed on PDAC tumors from 25 germline ATM PV carriers diagnosed at Mayo Clinic between 2007 and 2017. Somatic and copy-number alterations, mutational signatures, transcriptomic subtypes, and the immune landscape were evaluated. Results: High-quality whole-exome and whole-genome sequencing were obtained from 21 and 15 tumors, respectively. Biallelic inactivation of ATM was observed in 87%, KRAS PV in 90%, CDKN2A homozygous loss in 60%, and TP53 alterations in <10% of these tumors. A predominant clock-like mutational signature was present in all samples. Whole-transcriptome analysis identified that the aberrantly differentiated endocrine exocrine subtype accounted for 18% of PDAC and was consistently associated with >5-year overall survival. In addition, a 28-gene expression–based signature associated with overall survival was identified and further validated in The Cancer Genome Atlas cohort. Immune landscape analysis through CODEX identified enriched CD4 T-helper cell/tumor interactions and reduced B7H3-high cell/tumor interactions in ATM PV carriers compared with noncarriers. Conclusions: The observed absence of TP53 PV and enrichment for CDKN2A alterations in ATM tumors, along with differences in the mutational signatures, transcriptomic subtypes and immune landscape, improve our understanding of the mechanistic pathways involved in PDAC development in germline ATM PV carriers and help identify potential targeted therapeutic strategies.

Abstract Objective Despite exponential growth in artificial intelligence (AI) research for laboratory medicine and pathology, a significant gap exists between model development and clinical AI implementation. This article introduces a structured framework, the Clinical AI Readiness Evaluator (CARE), to bridge this gap and support the responsible adoption of AI in clinical laboratory settings. Methods Building upon the Machine Learning Technology Readiness Levels framework, we developed CARE specifically for the clinical laboratory environment by incorporating health care–specific requirements, regulatory considerations, and workflow integration needs. This framework was iteratively refined through practical application across diverse AI use cases within laboratory medicine and pathology. Results The CARE framework provides a systematic approach to AI development and implementation through 8 component workstreams: clinical use case, data, data pipeline, code, clinical user experience, clinical technology infrastructure, clinical orchestration, and regulatory compliance. Unlike generic AI frameworks, CARE distinguishes itself by emphasizing both health care and laboratory workflow integration, regulatory requirements, ethical considerations, and comprehensive validation for clinical contexts. The framework accommodates both internally developed models and commercial AI solutions, providing clear guidance through technology readiness levels and structured review processes. Conclusions The CARE framework addresses the unique challenges of implementing AI in laboratory medicine and pathology by providing a comprehensive roadmap from initial concepts through clinical deployment and maintenance. This article, the first in a series of 4, establishes the foundational AI lifecycle framework, while subsequent articles will explore data documentation, ethical AI considerations, and governance structures. By adopting this structured approach, laboratories can responsibly harness AI’s potential to enhance diagnostic accuracy and operational efficiencies and, ultimately, improve patient care.

Abstract Recent advances in the field of immuno‐oncology have brought transformative changes in the management of cancer patients. The immune profile of tumours has been found to have key value in predicting disease prognosis and treatment response in various cancers. Multiplex immunohistochemistry and immunofluorescence have emerged as potent tools for the simultaneous detection of multiple protein biomarkers in a single tissue section, thereby expanding opportunities for molecular and immune profiling while preserving tissue samples. By establishing the phenotype of individual tumour cells when distributed within a mixed cell population, the identification of clinically relevant biomarkers with high‐throughput multiplex immunophenotyping of tumour samples has great potential to guide appropriate treatment choices. Moreover, the emergence of novel multi‐marker imaging approaches can now provide unprecedented insights into the tumour microenvironment, including the potential interplay between various cell types. However, there are significant challenges to widespread integration of these technologies in daily research and clinical practice. This review addresses the challenges and potential solutions within a structured framework of action from a regulatory and clinical trial perspective. New developments within the field of immunophenotyping using multiplexed tissue imaging platforms and associated digital pathology are also described, with a specific focus on translational implications across different subtypes of cancer. © 2024 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland.

Abstract Prostate cancer risk is influenced by rare and common germline variants. We examined the aggregate association of rare germline pathogenic/likely pathogenic/deleterious (P/LP/D) variants in ATM, BRCA2, PALB2, and NBN with a polygenic risk score (PRS) on prostate cancer risk among 1,796 prostate cancer cases (222 metastatic) and 1,424 controls of African ancestry. Relative to P/LP/D non-carriers at average genetic risk (33%–66% of PRS), men with low (0%–33%) and high (66%–100%) PRS had Odds Ratios (ORs) for overall prostate cancer of 2.08 [95% confidence interval (CI) = 0.58–7.49] and 18.06 (95% CI = 4.24–76.84) among P/LP/D carriers and 0.57 (95% CI = 0.46–0.71) and 3.02 (95% CI = 2.53–3.60) among non-carriers, respectively. The OR for metastatic prostate cancer was 2.73 (95% CI = 0.24–30.54) and 28.99 (95% CI = 4.39–191.43) among P/LP/D carriers and 0.54 (95% CI = 0.31–0.95) and 3.22 (95% CI = 2.20–4.73) among non-carriers, for men with low and high PRS, respectively. Lifetime absolute risks of overall prostate cancer increased with PRS (low to high) from 9.8% to 51.5% in P/LP/D carriers and 5.5% to 23.9% in non-carriers. Lifetime absolute risks of metastatic prostate cancer increased with PRS from 1.9% to 18.1% in P/LP/D carriers and 0.3% to 2.2% in non-carriers These findings suggest that assessment of prostate cancer risk for rare variant carriers should include PRS status. Significance: These findings highlight the importance of considering rare and common variants to comprehensively assess prostate cancer risk in men of African ancestry.

AbstractQuantifying tumor‐infiltrating lymphocytes (TILs) in breast cancer tumors is a challenging task for pathologists. With the advent of whole slide imaging that digitizes glass slides, it is possible to apply computational models to quantify TILs for pathologists. Development of computational models requires significant time, expertise, consensus, and investment. To reduce this burden, we are preparing a dataset for developers to validate their models and a proposal to the Medical Device Development Tool (MDDT) program in the Center for Devices and Radiological Health of the U.S. Food and Drug Administration (FDA). If the FDA qualifies the dataset for its submitted context of use, model developers can use it in a regulatory submission within the qualified context of use without additional documentation. Our dataset aims at reducing the regulatory burden placed on developers of models that estimate the density of TILs and will allow head‐to‐head comparison of multiple computational models on the same data. In this paper, we discuss the MDDT preparation and submission process, including the feedback we received from our initial interactions with the FDA and propose how a qualified MDDT validation dataset could be a mechanism for open, fair, and consistent measures of computational model performance. Our experiences will help the community understand what the FDA considers relevant and appropriate (from the perspective of the submitter), at the early stages of the MDDT submission process, for validating stromal TIL density estimation models and other potential computational models. © 2023 The Authors. The Journal of Pathology published by John Wiley & Sons Ltd on behalf of The Pathological Society of Great Britain and Ireland. This article has been contributed to by U.S. Government employees and their work is in the public domain in the USA.

Abstract Modern histologic imaging platforms coupled with machine learning methods have provided new opportunities to map the spatial distribution of immune cells in the tumor microenvironment. However, there exists no standardized method for describing or analyzing spatial immune cell data, and most reported spatial analyses are rudimentary. In this review, we provide an overview of two approaches for reporting and analyzing spatial data (raster versus vector‐based). We then provide a compendium of spatial immune cell metrics that have been reported in the literature, summarizing prognostic associations in the context of a variety of cancers. We conclude by discussing two well‐described clinical biomarkers, the breast cancer stromal tumor infiltrating lymphocytes score and the colon cancer Immunoscore, and describe investigative opportunities to improve clinical utility of these spatial biomarkers. © 2023 The Pathological Society of Great Britain and Ireland.