Kahn, Charles E.

A comprehensive, machine-interpretable taxonomy of artificial intelligence (AI) performance metrics provides a shared vocabulary and structured foundation to facilitate rigorous, transparent, and responsible evaluation of medical AI systems.

Abstract Medical imaging is undergoing a transformation driven by the advent of new, highly effective, machine learning techniques paired with increases in computational capabilities (Cheng et al. 2021; Gilson et al. 2023; Almeida et al. 2024; Krishna et al. 2024). These advanced algorithms have the potential to improve disease detection, diagnosis, prognosis, and treatment outcomes. However, the complexity of machine learning models, the large amounts of curated and annotated data required by some methods, and the potential for bias and error make it challenging for individuals to safely and effectively leverage these methods (Lin et al. 2024; Guo et al. 2024; Xu et al. 2024; Linguraru et al. 2024; Wood et al. 2019). To address these challenges, the American Association of Physicists in Medicine (AAPM), American College of Radiology (ACR), Radiological Society of North America (RSNA), and Society for Imaging Informatics in Medicine (SIIM) have worked together to develop a syllabus detailing a recommended set of competencies for medical imaging professionals interacting with these systems. This guide is aimed at four different personas: users of AI systems, purchasers of AI systems, individuals who provide clinical expertise during the development of AI systems (“clinical collaborators”), and developers of AI systems.1 This is a syllabus, not a curriculum, and is intentional in this scope. Recognizing that individuals may benefit from different presentations of the same material, this work enumerates a series of relevant competencies but does not prescribe, nor offer, a method of instruction (Schuur, Rezazade Mehrizi, and Ranschaert 2021; Garin et al. 2023). By addressing the task‐specific demands of each role, this guide will enable medical imaging professionals to utilize machine learning systems more safely and effectively, ultimately improving patient care and outcomes.

Although generative artificial intelligence can create podcast summaries of radiology articles, summaries are limited by hallucinations and lack of audience appropriateness, requiring further investigation to justify use as an educational tool.

Abstract Objective Textual radiology reports contain a wealth of information that may help understand associations among diseases and imaging observations. This study evaluated the ability to detect causal associations among diseases and imaging findings from their co-occurrence in radiology reports. Materials and Methods This IRB-approved and HIPAA-compliant study analyzed 1 702 462 consecutive reports of 1 396 293 patients; patient consent was waived. Reports were analyzed for positive mention of 16 839 entities (disorders and imaging findings) of the Radiology Gamuts Ontology (RGO). Entities that occurred in fewer than 25 patients were excluded. A Bayesian network structure-learning algorithm was applied at P < 0.05 threshold: edges were evaluated as possible causal relationships. RGO and/or physician consensus served as ground truth. Results 2742 of 16 839 RGO entities were included, 53 849 patients (3.9%) had at least one included entity. The algorithm identified 725 pairs of entities as causally related; 634 were confirmed by reference to RGO or physician review (87% precision). As shown by its positive likelihood ratio, the algorithm increased detection of causally associated entities 6876-fold. Discussion Causal relationships among diseases and imaging findings can be detected with high precision from textual radiology reports. Conclusion This approach finds causal relationships among diseases and imaging findings with high precision from textual radiology reports, despite the fact that causally related entities represent only 0.039% of all pairs of entities. Applying this approach to larger report text corpora may help detect unspecified or heretofore unrecognized associations.

This study explored the ability to identify causal relationships between diseases and imaging findings from their co-occurrences in radiology reports. A natural language processing (NLP) system with negative-expression filtering detected positive mentions of 16,912 disorders, interventions, and imaging findings in 1,702,462 consecutive radiology reports; the 55,564 causal relations defined by the Radiology Gamuts Ontology (RGO) served as reference standard. Conditions were considered to co-occur if they were present in reports from the same patient. The ϕ and κ statistics both achieved AUC0.70, P<0.001 in identifying causal relationships from pairwise co-occurrence data. Analysis of radiology reports can identify a large proportion of known causal associations among diseases and imaging findings. Automated approaches hold promise to identify causal relationships among diseases and imaging findings from their co-occurrence in text-based radiology reports.