Baca, Monica E. de

Context.— Generative artificial intelligence (GAI) is a promising new technology with the potential to transform communication and workflows in health care and pathology. Although new technologies offer advantages, they also come with risks that users, particularly early adopters, must recognize. Given the fast pace of GAI developments, pathologists may find it challenging to stay current with the terminology, technical underpinnings, and latest advancements. Building this knowledge base will enable pathologists to grasp the potential risks and impacts that GAI may have on the future practice of pathology. Objective.— To present key elements of GAI development, evaluation, and implementation in a way that is accessible to pathologists and relevant to laboratory applications. Data Sources.— Information was gathered from recent studies and reviews from PubMed and arXiv. Conclusions.— GAI offers many potential benefits for practicing pathologists. However, the use of GAI in clinical practice requires rigorous oversight and continuous refinement to fully realize its potential and mitigate inherent risks. The performance of GAI is highly dependent on the quality and diversity of the training and fine-tuning data, which can also propagate biases if not carefully managed. Ethical concerns, particularly regarding patient privacy and autonomy, must be addressed to ensure responsible use. By harnessing these emergent technologies, pathologists will be well placed to continue forward as leaders in diagnostic medicine.

Context.— Machine learning applications in the pathology clinical domain are emerging rapidly. As decision support systems continue to mature, laboratories will increasingly need guidance to evaluate their performance in clinical practice. Currently there are no formal guidelines to assist pathology laboratories in verification and/or validation of such systems. These recommendations are being proposed for the evaluation of machine learning systems in the clinical practice of pathology. Objective.— To propose recommendations for performance evaluation of in vitro diagnostic tests on patient samples that incorporate machine learning as part of the preanalytical, analytical, or postanalytical phases of the laboratory workflow. Topics described include considerations for machine learning model evaluation including risk assessment, predeployment requirements, data sourcing and curation, verification and validation, change control management, human-computer interaction, practitioner training, and competency evaluation. Data Sources.— An expert panel performed a review of the literature, Clinical and Laboratory Standards Institute guidance, and laboratory and government regulatory frameworks. Conclusions.— Review of the literature and existing documents enabled the development of proposed recommendations. This white paper pertains to performance evaluation of machine learning systems intended to be implemented for clinical patient testing. Further studies with real-world clinical data are encouraged to support these proposed recommendations. Performance evaluation of machine learning models is critical to verification and/or validation of in vitro diagnostic tests using machine learning intended for clinical practice.

Abstract Background Laboratory medicine has reached the era where promises of artificial intelligence and machine learning (AI/ML) seem palpable. Currently, the primary responsibility for risk-benefit assessment in clinical practice resides with the medical director. Unfortunately, there is no tool or concept that enables diagnostic quality assessment for the various potential AI/ML applications. Specifically, we noted that an operational definition of laboratory diagnostic quality – for the specific purpose of assessing AI/ML improvements – is currently missing. Methods A session at the 3rd Strategic Conference of the European Federation of Laboratory Medicine in 2022 on “ AI in the Laboratory of the Future” prompted an expert roundtable discussion. Here we present a conceptual diagnostic quality framework for the specific purpose of assessing AI/ML implementations. Results The presented framework is termed diagnostic quality model (DQM) and distinguishes AI/ML improvements at the test, procedure, laboratory, or healthcare ecosystem level. The operational definition illustrates the nested relationship among these levels. The model can help to define relevant objectives for implementation and how levels come together to form coherent diagnostics. The affected levels are referred to as scope and we provide a rubric to quantify AI/ML improvements while complying with existing, mandated regulatory standards. We present 4 relevant clinical scenarios including multi-modal diagnostics and compare the model to existing quality management systems. Conclusions A diagnostic quality model is essential to navigate the complexities of clinical AI/ML implementations. The presented diagnostic quality framework can help to specify and communicate the key implications of AI/ML solutions in laboratory diagnostics.

Context.— Biomedical terminologies such as Logical Observation Identifiers, Names, and Codes (LOINC) were developed to enable interoperability of health care data between disparate health information systems to improve patient outcomes, public health, and research activities. Objective.— To ascertain the utilization rate and accuracy of LOINC terminology mapping to 10 commonly ordered tests by participants of the College of American Pathologists (CAP) Proficiency Testing program. Design.— Questionnaires were sent to 1916 US and Canadian laboratories participating in the 2018 CAP coagulation (CGL) and/or cardiac markers (CRT) surveys requesting information on practice setting, instrument(s) and test method(s), and LOINC code selection and usage in the laboratory and electronic health records. Results.— Ninety of 1916 CGL and/or CRT participants (4.7%) responded to the questionnaire. Of the 275 LOINC codes reported, 54 (19.6%) were incorrect: 2 codes (5934-2 and 12345-1) (0.7%) did not exist in the LOINC database and the highest error rates were observed in the property (27 of 275, 9.8%), system (27 of 275, 9.8%), and component (22 of 275, 8.0%) LOINC axes. Errors in LOINC code selection included selection of the incorrect component (eg, activated clotting time instead of activated partial thromboplastin time); selection of panels that can never be used to obtain an individual analyte (eg, prothrombin time panel instead of international normalized ratio); and selection of an incorrect specimen type. Conclusions.— These findings of real-world LOINC code implementation across a spectrum of laboratory settings should raise concern about the reliability and utility of using LOINC for clinical research or to aggregate data.