Zarella, Mark D.

Abstract Objectives Traditional laboratory regulatory frameworks provide robust guidance for conventional clinical testing but lack requirements for data documentation for artificial intelligence and machine learning (AI/ML) solutions. This gap creates significant risks since AI solutions directly inherit patterns, biases, and limitations from their training data. Here, we describe the development and implementation of a comprehensive data documentation checklist with accompanying data lineage for use within an AI lifecycle framework for clinical laboratories. Methods Building on the previously established Clinical AI Readiness Evaluator (CARE) framework, we developed a comprehensive data checklist and data lineage methodology through a multiphase process, including (1) comprehensive review of existing AI/ML data documentation frameworks, (2) focused meetings with 3 institutional AI operations teams, and (3) 3 rounds of iterative refinement by our multidisciplinary team. The checklist’s effectiveness was then assessed using 3 diverse AI/ML projects moving through the CARE framework. Results The CARE Data Checklist and Data Lineage provide a structured approach to documenting critical aspects of datasets used in AI/ML projects, including data composition, demographics, collection methods, labeling processes, usage constraints, maintenance requirements, and a data readiness assessment. The checklist addresses unique data-centric challenges of AI/ML applications, facilitating transparency, reproducibility, and regulatory compliance. Conclusions The CARE Data Checklist and Data Lineage serve as both a technical guide and a communication tool bridging gaps between technical and clinical stakeholders. By working on these documents early in the AI lifecycle, laboratories can anticipate and address data-related challenges, ultimately saving time, optimizing resources, and improving the reliability of AI-augmented laboratory solutions.

Abstract Objective Despite exponential growth in artificial intelligence (AI) research for laboratory medicine and pathology, a significant gap exists between model development and clinical AI implementation. This article introduces a structured framework, the Clinical AI Readiness Evaluator (CARE), to bridge this gap and support the responsible adoption of AI in clinical laboratory settings. Methods Building upon the Machine Learning Technology Readiness Levels framework, we developed CARE specifically for the clinical laboratory environment by incorporating health care–specific requirements, regulatory considerations, and workflow integration needs. This framework was iteratively refined through practical application across diverse AI use cases within laboratory medicine and pathology. Results The CARE framework provides a systematic approach to AI development and implementation through 8 component workstreams: clinical use case, data, data pipeline, code, clinical user experience, clinical technology infrastructure, clinical orchestration, and regulatory compliance. Unlike generic AI frameworks, CARE distinguishes itself by emphasizing both health care and laboratory workflow integration, regulatory requirements, ethical considerations, and comprehensive validation for clinical contexts. The framework accommodates both internally developed models and commercial AI solutions, providing clear guidance through technology readiness levels and structured review processes. Conclusions The CARE framework addresses the unique challenges of implementing AI in laboratory medicine and pathology by providing a comprehensive roadmap from initial concepts through clinical deployment and maintenance. This article, the first in a series of 4, establishes the foundational AI lifecycle framework, while subsequent articles will explore data documentation, ethical AI considerations, and governance structures. By adopting this structured approach, laboratories can responsibly harness AI’s potential to enhance diagnostic accuracy and operational efficiencies and, ultimately, improve patient care.

AbstractBreast cancer affects millions globally, necessitating precise biomarker testing for effective treatment. HER2 testing is crucial for guiding therapy, particularly with novel antibody‐drug conjugates (ADCs) like trastuzumab deruxtecan, which shows promise for breast cancers with low HER2 expression. Current HER2 testing methods, including immunohistochemistry (IHC) and in situ hybridization (ISH), have limitations. IHC, a semi‐quantitative assay, is prone to interobserver variability. While ISH provides higher precision than IHC, it remains more resource‐intensive in terms of cost and workflow. However, turnaround time is typically faster than that of other advanced molecular methods such as next‐generation sequencing. We adapted the clustering‐constrained‐attention multiple‐instance deep learning model to improve IHC testing and reduce dependence on reflex fluorescence ISH (FISH) tests. Using 5,731 HER2 IHC images, including 592 cases with FISH testing, we trained two models: one for predicting HER2 scores from IHC images and another for predicting FISH scores from equivocal cases. The HER2 IHC score prediction model achieved 91% ± 0.01 overall accuracy and a receiver operating characteristic (ROC) area under the curve (AUC) of 0.98 ± 0.01. The FISH score prediction model had an ROC AUC of 0.84 ± 0.07, with sensitivity at 0.37 ± 0.13 and specificity at 0.96 ± 0.03. External validation on cases from 203 institutions showed similar performance. The HER2 IHC model maintained a 91% ± 0.01 accuracy and an ROC AUC of 0.98 ± 0.01, while the FISH model had an ROC AUC of 0.75 ± 0.03, with sensitivity at 0.28 ± 0.04 and specificity at 0.93 ± 0.01. Our model advances HER2 scoring by reducing subjectivity and variability in current scoring methods. Despite lower accuracy and sensitivity in the FISH prediction model, it may be a beneficial option for settings where reflex FISH testing is unavailable or prohibitive. With high specificity, our model can serve as an effective screening tool, enhancing breast cancer diagnosis and treatment selection.

Context.— Computational pathology combines clinical pathology with computational analysis, aiming to enhance diagnostic capabilities and improve clinical productivity. However, communication barriers between pathologists and developers often hinder the full realization of this potential. Objective.— To propose a standardized framework that improves mutual understanding of clinical objectives and computational methodologies. The goal is to enhance the development and application of computer-aided diagnostic (CAD) tools. Design.— This article suggests pivotal roles for pathologists and computer scientists in the CAD development process. It calls for increased understanding of computational terminologies, processes, and limitations among pathologists. Similarly, it argues that computer scientists should better comprehend the true use cases of the developed algorithms to avoid clinically meaningless metrics. Results.— CAD tools improve pathology practice significantly. Some tools have even received US Food and Drug Administration approval. However, improved understanding of machine learning models among pathologists is essential to prevent misuse and misinterpretation. There is also a need for a more accurate representation of the algorithms’ performance compared to that of pathologists. Conclusions.— A comprehensive understanding of computational and clinical paradigms is crucial for overcoming the translational gap in computational pathology. This mutual comprehension will improve patient care through more accurate and efficient disease diagnosis.

Context.— Machine learning applications in the pathology clinical domain are emerging rapidly. As decision support systems continue to mature, laboratories will increasingly need guidance to evaluate their performance in clinical practice. Currently there are no formal guidelines to assist pathology laboratories in verification and/or validation of such systems. These recommendations are being proposed for the evaluation of machine learning systems in the clinical practice of pathology. Objective.— To propose recommendations for performance evaluation of in vitro diagnostic tests on patient samples that incorporate machine learning as part of the preanalytical, analytical, or postanalytical phases of the laboratory workflow. Topics described include considerations for machine learning model evaluation including risk assessment, predeployment requirements, data sourcing and curation, verification and validation, change control management, human-computer interaction, practitioner training, and competency evaluation. Data Sources.— An expert panel performed a review of the literature, Clinical and Laboratory Standards Institute guidance, and laboratory and government regulatory frameworks. Conclusions.— Review of the literature and existing documents enabled the development of proposed recommendations. This white paper pertains to performance evaluation of machine learning systems intended to be implemented for clinical patient testing. Further studies with real-world clinical data are encouraged to support these proposed recommendations. Performance evaluation of machine learning models is critical to verification and/or validation of in vitro diagnostic tests using machine learning intended for clinical practice.