Stram, Michelle N

Context.— Machine learning applications in the pathology clinical domain are emerging rapidly. As decision support systems continue to mature, laboratories will increasingly need guidance to evaluate their performance in clinical practice. Currently there are no formal guidelines to assist pathology laboratories in verification and/or validation of such systems. These recommendations are being proposed for the evaluation of machine learning systems in the clinical practice of pathology. Objective.— To propose recommendations for performance evaluation of in vitro diagnostic tests on patient samples that incorporate machine learning as part of the preanalytical, analytical, or postanalytical phases of the laboratory workflow. Topics described include considerations for machine learning model evaluation including risk assessment, predeployment requirements, data sourcing and curation, verification and validation, change control management, human-computer interaction, practitioner training, and competency evaluation. Data Sources.— An expert panel performed a review of the literature, Clinical and Laboratory Standards Institute guidance, and laboratory and government regulatory frameworks. Conclusions.— Review of the literature and existing documents enabled the development of proposed recommendations. This white paper pertains to performance evaluation of machine learning systems intended to be implemented for clinical patient testing. Further studies with real-world clinical data are encouraged to support these proposed recommendations. Performance evaluation of machine learning models is critical to verification and/or validation of in vitro diagnostic tests using machine learning intended for clinical practice.

PittUDT, a recursive partitioning decision tree algorithm for predicting urine culture (UC) positivity based on macroscopic and microscopic urinalysis (UA) parameters, was developed in support of a broader system-wide diagnostic stewardship initiative to increase appropriateness of UC testing. Reflex algorithm training utilized results from 19,511 paired UA and UC cases (26.8% UC positive); the average patient age was 57.4 years, and 70% of samples were from female patients.

Context.— Biomedical terminologies such as Logical Observation Identifiers, Names, and Codes (LOINC) were developed to enable interoperability of health care data between disparate health information systems to improve patient outcomes, public health, and research activities. Objective.— To ascertain the utilization rate and accuracy of LOINC terminology mapping to 10 commonly ordered tests by participants of the College of American Pathologists (CAP) Proficiency Testing program. Design.— Questionnaires were sent to 1916 US and Canadian laboratories participating in the 2018 CAP coagulation (CGL) and/or cardiac markers (CRT) surveys requesting information on practice setting, instrument(s) and test method(s), and LOINC code selection and usage in the laboratory and electronic health records. Results.— Ninety of 1916 CGL and/or CRT participants (4.7%) responded to the questionnaire. Of the 275 LOINC codes reported, 54 (19.6%) were incorrect: 2 codes (5934-2 and 12345-1) (0.7%) did not exist in the LOINC database and the highest error rates were observed in the property (27 of 275, 9.8%), system (27 of 275, 9.8%), and component (22 of 275, 8.0%) LOINC axes. Errors in LOINC code selection included selection of the incorrect component (eg, activated clotting time instead of activated partial thromboplastin time); selection of panels that can never be used to obtain an individual analyte (eg, prothrombin time panel instead of international normalized ratio); and selection of an incorrect specimen type. Conclusions.— These findings of real-world LOINC code implementation across a spectrum of laboratory settings should raise concern about the reliability and utility of using LOINC for clinical research or to aggregate data.