Kehl, Kenneth L.

PURPOSE To evaluate open-source large language models (LLMs) for extracting cancer-specific phenotypic data, benchmark their performance against GPT4 models, and assess the impact of fine-tuning with training data sizes. METHODS Open-source LLMs (Mistral, LLaMa, MAMBA, BioMistral) were evaluated in zero-/one-shot and fine-tuned setups against GPT4-turbo/GPT4o to extract the cancer presence, progression, response, and metastatic sites from radiology impressions of patients with solid tumors treated at Dana-Farber Cancer Institute. Performance metrics (accuracy, precision, recall, F1-score) were computed. McNemar's odds ratio (OR), measuring which model is more likely to be correct when they disagree, was computed with 95% CI. Statistical significance was assessed using the alpha of .000139. RESULTS This study included 2,623 patients (25,273 radiology impressions). In zero-/one-shot settings, GPT4-turbo/GPT4o outperformed open-source LLMs. However, fine-tuned open-source LLMs achieved higher F1-scores than GPT4 models. Compared with the best-performing GPT4 model, fine-tuned Mistral0.2-7.3B (OR, 0.27 [95% CI, 0.20 to 0.36]; P < .00001), Mistral0.3-7.3B (OR, 0.26 [95% CI, 0.19 to 0.36]; P < .00001), LLaMa2-6.7B (OR, 0.30 [95% CI, 0.22 to 0.40]; P < .00001), LLaMa3.1-8B (OR, 0.37 [95% CI, 0.28 to 0.48]; P < .00001), and MAMBA-2.8B (OR, 0.32 [95% CI, 0.24 to 0.42]; P < .00001) showed significantly better performance in ascertaining disease progression. Performance was consistently better for inferring overall response, any evidence of cancer, and sites of metastases, with no significant differences among fine-tuned open-source LLMs. Fine-tuning gains plateaued at 25% of training data (5,718 impressions) and remained comparable at 5% (1,144 impressions). CONCLUSION Open-source LLMs, when fine-tuned using labeled data, can effectively automate the ascertainment of key radiophenotypic variables using only the impression section of radiology reports, without the full report text. Their consistent performance in small training sets suggests that these models may provide a scalable approach for phenotypic characterization of patients with cancer in real-world clinical settings.

Abstract INTRODUCTION: Prostate cancer (PC) shows marked heterogeneity in clinical outcomes, from indolent to highly aggressive disease. Alterations in TP53, PTEN, RB1, and DNA damage repair genes are linked to poor patient outcomes, yet how the interaction and co-occurrence of these alterations impact prognosis remains unclear. We leveraged the AACR Project GENIE prostate cancer cohort to assess the prognostic impact of individual and concomitant gene alterations, focusing on key PC drivers (BRCA2, TP53, PTEN, RB1). METHODS: the study was conducted in two stages: (i) identification of candidate alterations at individual gene level through a systematic literature review, followed by analyzing the prevalence and co-occurrence of genes in the AACR Project GENIE Public Release (PR) dataset. (ii) Prognostic analysis of the pre-selected biomarkers in the AACR Project GENIE Biopharma Collaborative (BPC) cohort. Analyses were performed in three clinical settings (entire BPC cohort, n=1114; subgroup of patients with localized disease at diagnosis, n=837; and subgroup of patients with metastatic PC with a biopsy collected before or at the time of mPC, n=455) using two statistical approaches: univariate analysis (UA) and inverse probability weighting using propensity scores (PS). Cox models were applied to estimate overall survival hazard ratios (HRs). RESULTS: after integrating the findings from the systematic review with the analysis of the GENIE PR (n=5684), 107 individual genes and 122 co-occurring alterations were selected for study. In the GENIE BPC (n=1114 patients) 75% were M0 at diagnosis, and 67% presented metastatic PC at any time. Alteration prevalence was: TP53, 25%; PTEN, 15%; BRCA2, 4%; RB1, 4%; co-alterations included, PTEN-TP53 (6.4%), RB1-TP53 (2.1%), PTEN-RB1 (1.3%), and BRCA2-TP53 (1.2%). All four key genes showed a statistically significant prognostic impact in the entire cohort with UA (BRCA2, HR:1.9; TP53, HR:2.6; PTEN, HR:2.4; RB1, HR:4.1, all p&lt;0.001). Results were consistent in the subgroup analysis and when using PS. Additional genes such as FANCA, PIK3CA, and MYC were also associated with poor outcomes. Among co-alterations, BRCA2-TP53 (HR:2.0, p=0.05), RB1-TP53 (HR:3.6, p&lt;0.001), PTEN-TP53 (HR:2.4, p&lt;0.001); PTEN-RB1 (HR:5.5, p&lt;0.001), and PTEN-TP53-RB1 (HR:9.0, p&lt;0.001) were the strongest predictors of poor survival, with consistent effects across subgroups and analytical methods. Other combinations including MYC-TP53 (HR 2.2, p&lt;0.001) and PIK3CA-TP53 (HR 3.2, p&lt;0.001), also showed independent prognostic value. CONCLUSIONS: Genomic alterations in TP53, PTEN, RB1, and BRCA2, alone and in combination, independently associate with poor prognosis. Co-alteration patterns such as PTEN-TP53-RB1 define aggressive molecular subsets. This evidence supports integrating genomic interaction patterns for enhancing clinical risk models and guiding precision patient management. Furthermore, this study highlights the potential of AACR Project GENIE to enable data-driven development of biomarker-informed patient care strategies. Citation Format: Pablo Cresta Morgado, Victor Navarro, Guillermo Vilacampa, Haitham Alatoom, Manuel Ramos del Rio, Irbaz Riaz, Shawn Sweeney, Nikolaus Schultz, Ken Kehl, Gregory Riely, Wasim Abida, Katherine Panageas, Deborah Schrag, Philippe Bedard, Christine Micheel, Xindi Guo, Chelsea Nayan, Rodrigo Dientsmann, Joaquin Mateo. Genomic prognostic biomarkers in prostate cancer from the AACR GENIE cohort [abstract]. In: Proceedings of the AACR Special Conference in Cancer Research: Innovations in Prostate Cancer Research and Treatment; 2026 Jan 20-22; Philadelphia PA. Philadelphia (PA): AACR; Cancer Res 2026;86(2_Suppl):Abstract nr A013.

Abstract Objective Data extraction from the published literature is the most laborious step in conducting living systematic reviews (LSRs). We aim to build a generalizable, automated data extraction workflow leveraging large language models (LLMs) that mimics the real-world 2-reviewer process. Materials and Methods A dataset of 10 trials (22 publications) from a published LSR was used, focusing on 23 variables related to trial, population, and outcomes data. The dataset was split into prompt development (n = 5) and held-out test sets (n = 17). GPT-4-turbo and Claude-3-Opus were used for data extraction. Responses from the 2 LLMs were considered concordant if they were the same for a given variable. The discordant responses from each LLM were provided to the other LLM for cross-critique. Accuracy, ie, the total number of correct responses divided by the total number of responses, was computed to assess performance. Results In the prompt development set, 110 (96%) responses were concordant, achieving an accuracy of 0.99 against the gold standard. In the test set, 342 (87%) responses were concordant. The accuracy of the concordant responses was 0.94. The accuracy of the discordant responses was 0.41 for GPT-4-turbo and 0.50 for Claude-3-Opus. Of the 49 discordant responses, 25 (51%) became concordant after cross-critique, increasing accuracy to 0.76. Discussion Concordant responses by the LLMs are likely to be accurate. In instances of discordant responses, cross-critique can further increase the accuracy. Conclusion Large language models, when simulated in a collaborative, 2-reviewer workflow, can extract data with reasonable performance, enabling truly “living” systematic reviews.

Importance Artificial intelligence (AI) tools are rapidly integrating into cancer care. Understanding stakeholder views on ethical issues associated with the implementation of AI in oncology is critical to optimal deployment. Objective To evaluate oncologists’ views on the ethical domains of the use of AI in clinical care, including familiarity, predictions, explainability (the ability to explain how a result was determined), bias, deference, and responsibilities. Design, Setting, and Participants This cross-sectional, population-based survey study was conducted from November 15, 2022, to July 31, 2023, among 204 US-based oncologists identified using the National Plan &amp;amp; Provider Enumeration System. Main Outcomes and Measures The primary outcome was response to a question asking whether participants agreed or disagreed that patients need to provide informed consent for AI model use during cancer treatment decisions. Results Of 387 surveys, 204 were completed (response rate, 52.7%). Participants represented 37 states, 120 (63.7%) identified as male, 128 (62.7%) as non-Hispanic White, and 60 (29.4%) were from academic practices; 95 (46.6%) had received some education on AI use in health care, and 45.3% (92 of 203) reported familiarity with clinical decision models. Most participants (84.8% [173 of 204]) reported that AI-based clinical decision models needed to be explainable by oncologists to be used in the clinic; 23.0% (47 of 204) stated they also needed to be explainable by patients. Patient consent for AI model use during treatment decisions was supported by 81.4% of participants (166 of 204). When presented with a scenario in which an AI decision model selected a different treatment regimen than the oncologist planned to recommend, the most common response was to present both options and let the patient decide (36.8% [75 of 204]); respondents from academic settings were more likely than those from other settings to let the patient decide (OR, 2.56; 95% CI, 1.19-5.51). Most respondents (90.7% [185 of 204]) reported that AI developers were responsible for the medico-legal problems associated with AI use. Some agreed that this responsibility was shared by physicians (47.1% [96 of 204]) or hospitals (43.1% [88 of 204]). Finally, most respondents (76.5% [156 of 204]) agreed that oncologists should protect patients from biased AI tools, but only 27.9% (57 of 204) were confident in their ability to identify poorly representative AI models. Conclusions and Relevance In this cross-sectional survey study, few oncologists reported that patients needed to understand AI models, but most agreed that patients should consent to their use, and many tasked patients with choosing between physician- and AI-recommended treatment regimens. These findings suggest that the implementation of AI in oncology must include rigorous assessments of its effect on care decisions as well as decisional responsibility when problems related to AI use arise.

Introduction Artificial Intelligence (AI) is an emerging technology that promises advancements across blood cancer discovery and care delivery. There are already FDA-approved uses of AI in hematology and oncology but few data available about its use or utility. Concerns have been raised over AI's potential biases, “explainability,” and who is responsible for its use. Oncologists' familiarity with AI and perspectives on these issues are not well known and are critical to understanding the ethical deployment of AI in blood cancer care. Methods From January to July 2023, we performed a national cross-sectional survey of practicing oncologists. A draft survey instrument was developed by a team with content and methodological expertise. Iterative refinement occurred through cognitive debriefing with 5 practicing hematologist/oncologists. The final instrument included 24 questions covering domains of AI familiarity, predictions, explainability, and responsibilities. A random sample of US oncologists was identified using the National Plan & Provider Enumeration System. Paper surveys were mailed with $25 incentivizing gift cards. Reminder letters and phone calls were used for non-responders with an electronic survey option provided. Responses were analyzed descriptively. X2 testing assessed bivariate associations between responses and participant practice setting, with odds ratios (OR) reported. Results Of 399 mailed surveys, 12 were undeliverable and 201 were completed for a response rate of 51.9%. Question missingness was &lt;1%. Participants were 63.2% White, 16.9% Indian Asian, 10.0% Eastern Asian or Pacific Islander, and 4.5% Black; 6.0% were Hispanic. They represented 37 states. The majority were male (63.7%), 40-59 years old (54.2%), and had no specific training in AI (52.2%); 68.2% had at least 11 years in practice and 29.9% practiced at a primary academic hospital. Questions assessing overall familiarity with and predictions related to AI are shown in Table 1. Those at primary academic hospitals were more likely to be familiar with radiology models (OR 2.82, p&lt;0.001) but not pathology, prognostic, or clinical decision models (all p&gt;0.05). They were more likely to predict that AI would improve side effect management (OR 2.01, p=0.03) and end-of-life decision making (OR 2.07, p=0.02). High proportions of respondents thought that AI prognostic (67.1%) and clinical decision models (62.1%) needed to be explainable by clinicians and researchers to be used in clinic. When posed with a scenario where an FDA-approved AI model selects a different regimen than the oncologist planned to propose (Figure 1), the largest proportion reported they would present both options and let the patient decide (37.3%). Of those recommending a regimen, respondents from primary academic hospitals more frequently recommend the AI-generated regimen (56.1%), while respondents from elsewhere recommended their original regimen (69.0%; OR 2.85, p=0.007). Acceptability of some direct-to-patient AI model applications is shown in Table 1. A majority of respondents (90.5%) thought that AI developers should be responsible for medico-legal problems caused by AI use; 47.8% thought that physicians using the tool should be responsible. Though most (76.1%) agreed that it was oncologists' responsibility to protect patients from biased AI tools, few (27.3%) were confident in their ability to identify how representative the data used in an AI model was relative to the population with that cancer (including 66.7% of those who agreed it was oncologists' responsibility). More respondents thought that patients should need to consent to the use of AI tools in cancer treatment decisions (81.1%) than cancer diagnosis (55.7%); those from primary academic hospitals were less likely to respond that patients should consent for treatment or diagnosis (OR 0.43, p=0.02 and OR 0.44, p=0.009). Conclusions In this national survey of oncologists' views on AI, there was optimism about its impact on care with notable caveats related to end-of-life care, patient trust, and care disparities. Despite many approving use of AI models too complex for patients to explain, a substantial proportion deferred to patient choice when AI and physician recommendations differed. There was also dissonance between accepting responsibility to protect patients from bias and respondents' self-perceived ability to do so.

ImportanceProstate cancer (PCa) is marked by disparities in clinical outcomes by race, ethnicity, and age. Equitable enrollment in clinical trials is fundamental to promoting health equity.ObjectiveTo evaluate disparities in the inclusion of racial and ethnic minority groups and older adults across PCa clinical trials.Data SourcesMEDLINE, Embase, and ClinicalTrials.gov were searched to identify primary trial reports from each database's inception through February 2021. Global incidence in age subgroups and US population-based incidence in racial and ethnic subgroups were acquired from the Global Burden of Disease and Surveillance, Epidemiology, and End Results 21 incidence databases respectively.Study SelectionAll phase 2/3 randomized PCa clinical trials were eligible for age disparity analyses. Trials recruiting exclusively from the US were eligible for primary racial and ethnic disparity analyses.Data Extraction and SynthesisThis study was reported in accordance with Preferred Reporting Items for Systematic Reviews and Meta-analyses (PRISMA) reporting guidelines. Data were pooled using a random-effects model.Main Outcomes and MeasuresEnrollment incidence ratios (EIRs), trial proportions (TPs) of participants 65 years or older or members of a racial and ethnic subgroup divided by global incidence in the corresponding age group, or US population–based incidence in the corresponding racial and ethnic subgroup, were calculated. Meta-regression was used to explore associations between trial characteristics and EIRs and trends in EIRs during the past 3 decades.ResultsOf 9552 participants among trials reporting race, 954 (10.8%) were African American/Black, 80 (1.5%) were Asian/Pacific Islander, and 8518 (78.5) were White. Of 65 US trials, 45 (69.2%) reported race and only 9 (13.8%) reported data on all 5 US racial categories. Of 286 global trials, 75 (26.2%) reported the enrollment proportion of older adults. Outcomes by race and age were reported in 2 (3.1%) and 41 (15.0%) trials, respectively. Black (EIR, 0.70; 95% CI, 0.59-0.83) and Hispanic (EIR, 0.70; 95% CI, 0.59-0.83) patients were significantly underrepresented in US trials. There was no disparity in older adult representation (TP, 21 143 [71.1%]; EIR, 1.00; 95% CI, 0.95-1.05). The representation of Black patients was lower in larger trials (meta-regression coefficient, −0.06; 95% CI, −0.10 to −0.02; P = .002).Conclusions and RelevanceThe results of this meta-analysis suggest that Black and Hispanic men are underrepresented in trials compared with their share of PCa incidence. The representation of Black patients has consistently remained low during the past 2 decades.

Abstract Defining patient-to-patient similarity is essential for the development of precision medicine in clinical care and research. Conceptually, the identification of similar patient cohorts appears straightforward; however, universally accepted definitions remain elusive. Simultaneously, an explosion of vendors and published algorithms have emerged and all provide varied levels of functionality in identifying patient similarity categories. To provide clarity and a common framework for patient similarity, a workshop at the American Medical Informatics Association 2019 Annual Meeting was convened. This workshop included invited discussants from academics, the biotechnology industry, the FDA, and private practice oncology groups. Drawing from a broad range of backgrounds, workshop participants were able to coalesce around 4 major patient similarity classes: (1) feature, (2) outcome, (3) exposure, and (4) mixed-class. This perspective expands into these 4 subtypes more critically and offers the medical informatics community a means of communicating their work on this important topic.