Dyson, Gregory

e21598 Background: While large language models (LLMs) like GPT-4 (OpenAI) and Claude Opus (Anthropic) perform well on general medical exams, limited data exist on their application in oncology, particularly in multiple-choice question (MCQ) scenarios. This study benchmarks the accuracy and assesses the clinical reasoning of LLMs on cutaneous oncology MCQs from the American Society of Clinical Oncology (ASCO) question bank. Methods: Skin oncology MCQs from the ASCO question bank were tested using GPT-4 (OpenAI) and Claude Opus (Anthropic). Questions were entered into OpenAI and Anthropic API playgrounds without additional prompting, with identical settings for both models (Temperature = 0, Token = Max). Each question was tested three times per model to assess response consistency and precision. Initial responses were collected, and chain-of-thought (COT) prompting was applied to encourage stepwise reasoning mimicking an oncologist’s thought process. The percentage of revised answers after COT prompting was analyzed. Accuracy before and after COT prompting was compared. Incorrect responses were reviewed by board-certified oncologists specializing in skin cancer at National Cancer Institute (NCI)-designated centers. Each oncologist scored the responses on clarity, clinical reasoning, bias, and confidence in the models’ utility in clinical practice. Qualitative feedback was collected via free-text comments and analyzed descriptively. Results: Of 66 skin oncology MCQs, Claude correctly answered 55 questions (83% accuracy, 95% CI: 72%–91%), while ChatGPT correctly answered 53 (80% accuracy, 95% CI: 68%–89%). Both models performed comparably, with overlapping confidence intervals. A chi-square test showed no significant difference between their performances (p = 0.65). After COT prompting, both tools remained at their respective accuracy level. Thematic analysis of incorrect answers revealed that LLM responses frequently deviated from clinical guidelines and failed to reflect evidence-based oncology practice, often neglecting key trial data like DREAMSeq and frontline therapeutic strategies. The most frequent errors were Treatment Planning Errors, where recommendations misaligned with NCCN guidelines, and Interpretation Errors, involving misinterpretation of clinical trials or patient data. Anchoring bias emerged as a recurring issue, with models fixating on details (e.g., BRAF mutation status) at the expense of broader reasoning. Conclusions: This study demonstrates that while large language models like GPT-4 and Claude Opus show promise in the field of oncology, they still face significant challenges in clinical reasoning and adherence to evidence-based practices. Continued refinement and training on specialized medical data are essential before these models can be reliably integrated into clinical workflows.

e13637 Background: Large language models (LLMs) like GPT-4 (OpenAI) and Claude Opus (Anthropic) showed high accuracy in medical multiple-choice exams, but data on their oncology-specific clinical reasoning and performance is limited. This study evaluates their accuracy and clinical reasoning on breast oncology multiple-choice questions (MCQs) from the American Society of Clinical Oncology (ASCO) question bank. Methods: Using OpenAI and Anthropic Application Programming Interface (APIs), questions were tested without additional prompts under consistent settings (GPT-4 and Claude Opus; Temperature = 0, Tokens = Max). Each question was tested three times to assess precision. Then, Chain-of-thought (COT) prompting was applied to promote LLMs stepwise reasoning to increase their accuracy. Accuracy before and after COT prompting was compared. Incorrect responses were reviewed by board-certified medical oncologists speclized in breast cancer, who scored reasoning clarity, bias, and clinical relevance. Qualitative feedback was descriptively analyzed. Results: A total of 273 breast oncology MCQs were evaluated across the two LLMs. GPT-4 achieved an initial accuracy of 81.3% (222/273; 95% CI: 76.3%-85.5%), compared to Claude Opus, which achieved an accuracy of 79.5% (217/273; 95% CI: 74.3%-83.9%). The Chi-squared test for difference between the models before chain-of-thought (COT) prompting yielded a p-value of 0.59, indicating no statistically significant difference in accuracy between GPT-4 and Claude Opus prior to COT prompting. After COT prompting, the performance of the two models diverged significantly. GPT-4 saw a net decline in accuracy, decreasing by a net of 1 correct answer, resulting in an overall accuracy of 80.95% (221/273). In contrast, Claude Opus experienced a notable improvement, with a net of 19 additional correct answers, leading to an accuracy of 86.4% (236/273). A statistical analysis using a Chi-squared test revealed a difference in accuracy between the two models of 5.5% (p = 0.08), demonstrating that the improvement in accuracy for Claude Opus after COT prompting was borderline statistically significant compared to GPT-4. Thematic analysis of oncologists' feedback revealed that the most common reasons for incorrect answers were reliance on outdated guidelines, misinterpretation of clinical trial data, and failure to consider multidisciplinary or patient-specific approaches in clinical decision-making. Conclusions: Although AI models can achieve high scores on multiple-choice exams, they still require human supervision. These models rely on potentially outdated training data and lack the ability to individualize patient care or apply data from clinical trials. especially in unique or unconventional/non textbook scenarios.

e20011 Background: The therapeutic landscape of lung cancer is rapidly evolving, presenting oncologists with the challenge of staying updated amidst an overwhelming influx of data. Clinical decision support (CDS) tools, including artificial intelligence (AI) and large language models (LLMs), may help bridge this gap. Evaluating the accuracy of LLMs in complex, real-world oncology scenarios is crucial to understanding their potential. Methods: Twenty-five de-identified lung cancer cases from the fellows’ clinic at Karmanos Cancer Institute, Detroit, MI, were analyzed. Two LLMs, GPT-4 (OpenAI) and Claude Opus (Anthropic), were assessed using advanced prompting techniques like persona-based and chain-of-thought prompting. Five board-certified lung cancer oncologists from NCI-designated centers evaluated LLM-generated responses based on accuracy, treatment recommendation comprehensiveness, and supportive care planning, using a 1–5 scale. Novel insights, the presence of fabricated information, and harmful recommendations were flagged as binary outcomes. Oncologists were blinded to the LLM source and actual treatment decisions. Results: Table 1 presents patient characteristics. GPT-4 achieved an average accuracy score of 4.2 (95% CI, 3.9–4.4), with 3.7 for comprehensiveness of medical/surgical treatment recommendations and 3.7 for supportive care planning. Six responses (32%) were flagged as potentially harmful, and two (8%) contained inaccuracies. Sixteen GPT-4 responses (64%) were rated trustworthy as a CDS tool. Claude Opus had an average accuracy score of 3.6 (95% CI, 3.1–4.1), scoring 3.6 for treatment recommendation comprehensiveness and 3.5 for supportive care planning. Nine responses (36%) were flagged for potential harm, and five (20%) included inaccuracies. Eleven Claude responses (44%) were deemed trustworthy. Significant differences were observed in accuracy (p=0.04) and trustworthiness (p=0.03) between models using McNemar's test. Other factors showed no statistical significance. Conclusions: GPT-4 outperformed Claude Opus in accuracy and trustworthiness, but both models demonstrated limitations, including harmful recommendations and inaccuracies. These findings highlight the need for improved LLM refinement before routine use as CDS tools in lung cancer treatment. Patient demographics and clinical characteristics. Category subcategory Number Median Age (range)- Yr 65 (26-78) Female 7 Male 18 Histology Adenocarcinoma 10 Squamous Cell Carcinoma (SCC) 7 Small Cell Carcinoma 6 Poorly Differentiated 2 Total 25 Stage NSCLC Stage 3 7 NSCLC Stage 4 13 Small Cell limited stage 3 Small Cell Extensive Stage 2

Abstract Purpose: Pancreatic ductal adenocarcinoma (PDAC) remains a deadly disease urgently requiring new treatments. Overexpression of the protein transporter exportin-1 (XPO1) leads to mislocalization of tumor-suppressor proteins (TSP) and their inactivation. Earlier, we showed that blocking XPO1 by CRISPR/Cas9 validated Selective Inhibitor of Nuclear Export (SINE) compounds (selinexor and analogs) restores the antitumor activity of multiple TSPs leading to suppression of PDAC in vitro and in orthotopic models. Experimental Design: We evaluate the synergy between SINE compounds and standard-of-care treatments in preclinical models and in a PDAC Phase Ib trial. Results: SINE compounds synergize with gemcitabine (GEM) and nanoparticle albumin–bound (nab)-paclitaxel leading to suppression of PDAC cellular growth and cancer stem cell (CSC) spheroids disintegration. Label-free quantitative proteome profiling with nuclear and cytoplasmic enrichment showed superior enhancement in nuclear protein fraction in combination treatment. Selinexor inhibited the growth of PDAC CSC and two patient-derived (PDX) subcutaneous xenografts. Selinexor–GEM–nab-paclitaxel blocked PDX and orthotopic tumor growth. In a phase 1b study (NCT02178436), 9 patients were exposed to selinexor (60 mg oral) with GEM (1,000 mg/m2 i.v.) and nab-paclitaxel (125 mg/m2 i.v.) on days 1, 8, and 15 of 28-day cycle. Two patients showed partial response, and 2 had stable disease. An outstanding, durable objective response was observed in one of the responders with progression-free survival of 16 months and overall survival of 22 months. Conclusions: Our preclinical and ongoing clinical study lends support to the use of selinexor–GEM–nab-paclitaxel as an effective therapy for metastatic PDAC.

Abstract Benign breast disease, a known risk factor for subsequent breast cancer, includes a histological spectrum of lesions that can be subdivided based on the overall impression into non-proliferative lesions, proliferative lesions without atypia, and atypical hyperplasia. Yet little is known about the molecular characteristics of benign breast tissue from African American women, who suffer from disproportionately high breast cancer mortality rates. 72 breast samples from African American women were obtained. RNA was extracted from archival formalin-fixed, paraffin-embedded (FFPE) blocks. Gene expression profiling was performed on these samples using the Affymetrix HTA2.0 exon array chip. A fit linear model using generalized least squares identified probe sets with differential expression in benign-in situ and benign-invasive breast cancer paired lesions. 36 benign and 36 breast cancer paired lesions were included in the study. In the benign-in situ set (n=20), 978 probe sets were differentially expressed (threshold: fold-change, 1.50; p < 0.05). Probe sets mapped to 125 annotated genes, of which 69 (55.2%) were involved in cancer. The benign-invasive set (n=52) identified 351 differentially expressed probe sets that mapped to 84 annotated genes, of which 61 (72.6%) had known cancer significance. Genes involved in cellular growth and proliferation were significantly over-expressed in both benign-in situ and benign-invasive lesions (p-values < 0.0104). In the population of 36 paired samples, differentially expressed genes were found to regulate the estrogen receptor (ESR1) through distinct network associations. Many genes are differentially expressed by benign and breast cancer lesions, and in situ/invasive histology contribute to the distinct molecular characteristics and pathways of breast malignancy. Our findings highlight the first study to assess expression profiling of paired benign and breast cancer lesions among African American women. Citation Format: Andreana Natalie Holowatyj, Julie J. Ruterbusch, Rouba Ali-Fehmi, Sudeshna Bandyopadhyay, Christine Mehner, Melody Stallings Mann, Gregory Dyson, Derek Radisky, Michele L. Cote. Expression profiling of paired benign and breast cancer lesions in African American women [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2017; 2017 Apr 1-5; Washington, DC. Philadelphia (PA): AACR; Cancer Res 2017;77(13 Suppl):Abstract nr 4253. doi:10.1158/1538-7445.AM2017-4253