Day, Patrick

Abstract Objectives Traditional laboratory regulatory frameworks provide robust guidance for conventional clinical testing but lack requirements for data documentation for artificial intelligence and machine learning (AI/ML) solutions. This gap creates significant risks since AI solutions directly inherit patterns, biases, and limitations from their training data. Here, we describe the development and implementation of a comprehensive data documentation checklist with accompanying data lineage for use within an AI lifecycle framework for clinical laboratories. Methods Building on the previously established Clinical AI Readiness Evaluator (CARE) framework, we developed a comprehensive data checklist and data lineage methodology through a multiphase process, including (1) comprehensive review of existing AI/ML data documentation frameworks, (2) focused meetings with 3 institutional AI operations teams, and (3) 3 rounds of iterative refinement by our multidisciplinary team. The checklist’s effectiveness was then assessed using 3 diverse AI/ML projects moving through the CARE framework. Results The CARE Data Checklist and Data Lineage provide a structured approach to documenting critical aspects of datasets used in AI/ML projects, including data composition, demographics, collection methods, labeling processes, usage constraints, maintenance requirements, and a data readiness assessment. The checklist addresses unique data-centric challenges of AI/ML applications, facilitating transparency, reproducibility, and regulatory compliance. Conclusions The CARE Data Checklist and Data Lineage serve as both a technical guide and a communication tool bridging gaps between technical and clinical stakeholders. By working on these documents early in the AI lifecycle, laboratories can anticipate and address data-related challenges, ultimately saving time, optimizing resources, and improving the reliability of AI-augmented laboratory solutions.

Abstract Background Artificial intelligence (AI) augmented laboratory tests can improve quality, efficiency, cost-effectiveness, and staff satisfaction. However, the clinical success of these tests can introduce unforeseen challenges for model retraining. This study describes the discovery of an “AI label feedback loop” in a clinically implemented AI-augmented kidney stone composition test. Methods An AI-augmented kidney stone composition test (V1) has been previously deployed for clinical kidney stone characterization. After several years of clinical use, a retrained model (V2) was developed using 6 times more data. Model performance of both V1 and V2 were evaluated across 3 datasets: a recent production validation (hold-out) set (mostly AI-influenced labels), the original V1 validation set (pre-AI, entirely human-labeled), and a subset of recent cases with exclusively human-generated or human-corrected labels. Results V2 demonstrated a 10% lower concordance rate than V1 when evaluated on the recent production hold-out set, despite a much larger training dataset. Performance between V1 and V2 was similar when applied to the pre-AI validation set. Notably, V2 outperformed V1 on the recent subset of cases with human-only or human-corrected labels, particularly for less-common stone types. These findings revealed an AI label feedback loop, confounding retraining and evaluation. Conclusions The integration of AI into clinical practice can potentially influence reported test results, complicating the development and evaluation of future models. To mitigate AI label feedback loops, ongoing human annotation and careful validation set construction are essential. These strategies can ensure reliable performance assessment and support the safe evolution of clinical AI systems.

Abstract Objective Despite exponential growth in artificial intelligence (AI) research for laboratory medicine and pathology, a significant gap exists between model development and clinical AI implementation. This article introduces a structured framework, the Clinical AI Readiness Evaluator (CARE), to bridge this gap and support the responsible adoption of AI in clinical laboratory settings. Methods Building upon the Machine Learning Technology Readiness Levels framework, we developed CARE specifically for the clinical laboratory environment by incorporating health care–specific requirements, regulatory considerations, and workflow integration needs. This framework was iteratively refined through practical application across diverse AI use cases within laboratory medicine and pathology. Results The CARE framework provides a systematic approach to AI development and implementation through 8 component workstreams: clinical use case, data, data pipeline, code, clinical user experience, clinical technology infrastructure, clinical orchestration, and regulatory compliance. Unlike generic AI frameworks, CARE distinguishes itself by emphasizing both health care and laboratory workflow integration, regulatory requirements, ethical considerations, and comprehensive validation for clinical contexts. The framework accommodates both internally developed models and commercial AI solutions, providing clear guidance through technology readiness levels and structured review processes. Conclusions The CARE framework addresses the unique challenges of implementing AI in laboratory medicine and pathology by providing a comprehensive roadmap from initial concepts through clinical deployment and maintenance. This article, the first in a series of 4, establishes the foundational AI lifecycle framework, while subsequent articles will explore data documentation, ethical AI considerations, and governance structures. By adopting this structured approach, laboratories can responsibly harness AI’s potential to enhance diagnostic accuracy and operational efficiencies and, ultimately, improve patient care.

Abstract Metals and metalloids including cobalt, gadolinium, lutetium, and germanium are used in numerous medical applications spanning diverse specialities including orthopedics, radiology, oncology, and healthcare artificial intelligence. These medical advances include cobalt containing orthopedic implants, gadolinium-based contrast agents, lutetium-containing cancer drugs, and germanium-based semiconductors. While these metal and metalloid-based solutions do improve patient care, there is a heavy side to how the elements needed for these solutions are mined, extracted, and discarded. These practices often exploit and harm vulnerable populations and environments. As healthcare professionals, we should be aware of the entire mineral to medicine lifecycle. As providers and consumers of these metal and metalloid-based solutions, we must advocate for more responsible and ethical extraction and recycling practices. As researchers and educators, we must promote and support continued research and development into less resource-intense medical solutions that can both improve patient care and sustainability.

Abstract Background We sought to evaluate key performance indicators related to an internally developed and deployed artificial intelligence (AI)-augmented kidney stone composition test system for potential improvements in test quality, efficiency, cost-effectiveness, and staff satisfaction. Methods We compared quality, efficiency, staff satisfaction, and financial data from the 6 months after the AI-augmented laboratory test system was deployed (test period) with data from the same 6-month period in the previous year (control period) to determine if AI-augmentation improved key performance indicators of this laboratory test. Results In the 6 months following the deployment (test period) of the AI-augmented kidney stone composition test system, 44 830 kidney stones were analyzed. Of these, 92% of kidney stones were eligible for AI-assisted interpretation. Out of these AI-eligible stones, 45% were able to be auto-released by the AI-augmented test system without human secondary review. Furthermore, the new AI-augmented kidney stone test system resulted in an apparent 40% reduction in incorrect laboratory results. Additionally, the new AI-augmented test system improved laboratory efficiency by 20%, improved staff satisfaction, and reduced the average analysis cost per kidney stone by $0.23. Conclusions The AI-augmented test system improved test quality, efficiency, cost-effectiveness and staff satisfaction related to this kidney stone composition test.

Abstract Background A dual filtration-based method for determination of serum labile bound copper (LBC) and LBC fraction (LBC/total copper) was developed. Reduced total copper, elevated LBC, and elevated LBC fraction have been reported in Wilson disease (WD). Methods To evaluate the diagnostic performance of these markers, samples were obtained from 21 WD treatment-naïve (WD-TN, no WD treatment or <28 days of treatment) patients, 46 WD standard-of-care–treated (WD-SOC) patients, along with 246 patients representing other potential disorders of copper status. These were then compared to 213 reference interval population patients. Results Receiver operating characteristic curves for the reference population vs WD-TN yielded areas under the curve for total copper, LBC, and LBC fraction, of 0.99, 0.81, and 0.98, respectively. Using Youden cutoffs, sensitivity/specificity for WD-TN was 95%/97% for total copper, 71%/85% for LBC, and 95%/94% for LBC fraction. LBC values, but not total copper and LBC fraction, differed substantially between WD-TN and WD-SOC cohorts. We propose a dual model wherein total copper and LBC fraction results must agree to be classified as a “positive” or “negative” result for WD. This correctly classified 19/21 WD-TN patients as positive, and 194/213 reference interval patients as negative. The remaining “indeterminate” patients (representing approximately 9% of the reference and the WD-TN populations) exhibited conflicting total copper and LBC fraction results. When indeterminate results are excluded, this model exhibited apparent 100% sensitivity/specificity. Conclusions Agreement of total serum copper and LBC fraction classification may constitute an effective “rule-in” and “rule-out” assessment for WD-TN patients.

Context.— Clinical testing for Wilson disease (WD) is potentially challenging. Measuring the fraction of labile bound copper (LBC) to total copper may be a promising alternative diagnostic tool with better sensitivity and specificity than some current biomarker approaches. A dual filtration–based inductively coupled mass spectrometry (ICP-MS) assay to measure LBC in serum was developed. Objective.— To establish a reference interval for LBC and LBC to total copper (LBC fraction) in a healthy adult population, and to examine associations between total copper, LBC, and LBC fraction with age, sex, menopausal status, hormone replacement therapy, and supplement use. Design.— Serum samples were collected from healthy male (n = 110) and female (n = 104) patients between the ages of 19 and 80 years. Total copper and LBC were analyzed using ICP-MS. Results were used to calculate the LBC fraction. Reference intervals were calculated for the 2.5th and 97.5th percentiles for both LBC and LBC fraction. Results.— The reference intervals for LBC were determined to be 13 to 105 ng/mL and 12 to 107 ng/mL for female and male patients, respectively. The reference intervals for the LBC fraction were 1.0% to 8.1% and 1.2% to 10.5% for female and male patients, respectively. No significant associations were found regarding age, menopausal status, hormone replacement therapy, or vitamin and supplement use. Conclusions.— Sex-specific reference intervals have now been established for LBC and LBC fraction. These data in conjunction with further testing of WD populations can be used to assess the sensitivity and specificity of LBC fraction in screening, monitoring, and diagnosis.

Abstract Background Analytical and post-analytical errors are a persistent problem, which can be benchmarked using sigma metrics. Manual processes are associated with error rates of 3–4 sigma. Secondary review of manual reporting is meant to reduce/eliminate errors and is widely used, yet the true efficacy of manual secondary review is difficult to assess. Kidney stone analysis is a manually intensive process, requiring personnel to identify stone composition from spectra and to transcribe stone percent composition into an LIS system. Recently, we developed an AI program that allows for quantitative determination of kidney stone composition from spectra. This allows for a unique opportunity to estimate error rates associated with manual reporting and those following secondary review by means of subsequent AI quality assurance analysis. Methods Experienced technologists assessed stone composition and estimated the percent of each component (± 10%), based on Fourier transform infrared (FTIR) spectroscopy. These manual spectral composition assessments are secondarily reviewed by other experienced technologists for identity and percentage composition accuracy. The recently described Kidney stone AI spectral analysis (RokkStar) was implemented to assess reported stone composition identity and percent composition for agreement following secondary review in 159 334 stone reports. AI flagged discrepant reports were reviewed by personnel and reports were revised when appropriate. Secondary reviewer error detection rate was obtained by monitoring detection in a subset of 7440 kidney stones, and was that rate was extrapolated to establish secondary review error detection rate over the entire data set. The sum (total error) of estimated secondary reviewer error detection, and well as AI-mediated error detection, and all client reported errors were used to determine error rates in sigma following manual reporting, secondary review, and AI quality assurance. Results Manual entry led to an estimated 2060 errors in 159 334 stone reports, yielding an initial sigma value of 3.73. Secondary manual review resulted in an estimated 1909 (error detection rate of 92.5%) of these errors being detected over the entire report set, improving the sigma value to 4.61. Post-verification AI caught an additional 138 errors, leading to an improved post-AI sigma value of 5.27. Post-AI quality assurance, there were 13 remaining errors, 9 of which were manual reporting process deviations subsequently caught by laboratory staff, and 4 were reporting errors observed by client physicians. All 13 of these errors involved usual stone constituents or typographical errors that were not assessed by the current AI quality assurance version. Conclusion This study represents a unique large-scale estimate of efficacy of secondary result review by means of AI quality assurance review. While the presence of undetected errors cannot be completely ruled out, in this case the relative paucity of client reported errors, as well the alignment of the estimated rate of manual errors with previous reports may indicate that most errors were detected in this system. Importantly, manual secondary review improved errors rates by ∼0.9 sigma, and the use of AI further improved estimated overall sigma metric by ∼0.7, leading to a “world class” 5+ sigma level for this complex reporting process.

Abstract Background While clinical tests are available to aid in the diagnosis of Wilson’s Disease (WD), some such as total copper determination lack sensitivity and specificity, while others such as liver tissue copper assessment are costly and invasive. WD affects copper metabolism and is known to cause low serum copper concentrations. In addition, an elevated percentage of total serum copper exists as non-protein bound or labile bound copper (LBC). Previous studies have shown that measuring the ratio of labile bound copper to total serum copper may aid in the diagnosis of WD. To support the utility of a laboratory developed test using serum LBC ratios to aid in the diagnosis of WD, serum samples from healthy donors were used to establish a reference range for LBC and the ratio of LBC to total copper in adults. Methods Serum samples were collected in trace metal free tubes from 214 donors (n = 110 male, n = 104 female) between the ages of 19–80 years old, selected against pre-determined exclusion criteria including acute hepatitis, chronic hepatitis, renal failure, biliary cirrhosis, celiac disease, and leukemia. Prior to analysis, samples went through a centrifugation-based preparation process to isolate the LBC fraction using molecular weight size exclusion filters and EDTA chelation. Samples were then analyzed via ICP-MS in kinetic energy discrimination (KED) mode to directly measure the LBC and total copper concentrations. LBC ratio values were determined from the LBC/total copper concentrations. Results Preliminary reference ranges were developed for both LBC and the LBC ratio based on the 2.5th and 97.5th percentiles with 95% confidence intervals. Established sex specific reference intervals for total copper were deemed acceptable per CLSI guidelines in document EP28-A3c, with ≤10% of results from this sample set falling outside of the established intervals. No statistically significant relationship was found in the mid-95th percentile between LBC concentration or ratio with age or sex. The reference intervals for LBC concentration were calculated as 13.41–104.66 and 11.69–106.50 ng/mL for females and males respectively. The reference intervals for LBC ratio were calculated as 1.04–8.12 and 1.17–10.48 for females and males respectively. Conclusions A reference range of LBC and LBC ratio has now been determined for the healthy adult population. Further testing can now be done on disease state populations to determine the sensitivity and specificity of LBC ratio in the detection, monitoring, or diagnosis of WD.