Arjyal, Lubina

e13637 Background: Large language models (LLMs) like GPT-4 (OpenAI) and Claude Opus (Anthropic) showed high accuracy in medical multiple-choice exams, but data on their oncology-specific clinical reasoning and performance is limited. This study evaluates their accuracy and clinical reasoning on breast oncology multiple-choice questions (MCQs) from the American Society of Clinical Oncology (ASCO) question bank. Methods: Using OpenAI and Anthropic Application Programming Interface (APIs), questions were tested without additional prompts under consistent settings (GPT-4 and Claude Opus; Temperature = 0, Tokens = Max). Each question was tested three times to assess precision. Then, Chain-of-thought (COT) prompting was applied to promote LLMs stepwise reasoning to increase their accuracy. Accuracy before and after COT prompting was compared. Incorrect responses were reviewed by board-certified medical oncologists speclized in breast cancer, who scored reasoning clarity, bias, and clinical relevance. Qualitative feedback was descriptively analyzed. Results: A total of 273 breast oncology MCQs were evaluated across the two LLMs. GPT-4 achieved an initial accuracy of 81.3% (222/273; 95% CI: 76.3%-85.5%), compared to Claude Opus, which achieved an accuracy of 79.5% (217/273; 95% CI: 74.3%-83.9%). The Chi-squared test for difference between the models before chain-of-thought (COT) prompting yielded a p-value of 0.59, indicating no statistically significant difference in accuracy between GPT-4 and Claude Opus prior to COT prompting. After COT prompting, the performance of the two models diverged significantly. GPT-4 saw a net decline in accuracy, decreasing by a net of 1 correct answer, resulting in an overall accuracy of 80.95% (221/273). In contrast, Claude Opus experienced a notable improvement, with a net of 19 additional correct answers, leading to an accuracy of 86.4% (236/273). A statistical analysis using a Chi-squared test revealed a difference in accuracy between the two models of 5.5% (p = 0.08), demonstrating that the improvement in accuracy for Claude Opus after COT prompting was borderline statistically significant compared to GPT-4. Thematic analysis of oncologists' feedback revealed that the most common reasons for incorrect answers were reliance on outdated guidelines, misinterpretation of clinical trial data, and failure to consider multidisciplinary or patient-specific approaches in clinical decision-making. Conclusions: Although AI models can achieve high scores on multiple-choice exams, they still require human supervision. These models rely on potentially outdated training data and lack the ability to individualize patient care or apply data from clinical trials. especially in unique or unconventional/non textbook scenarios.

9022 Background: Lobectomy is the current standard of care for patients with stage I non-small cell lung cancer (NSCLC). There is a lack of prospective data on the benefit of adjuvant chemotherapy (CT) in patients with negative margins but with high-risk features: lympho-vascular invasion (LVI) or visceral pleural invasion (VPI). We aimed to investigate the benefit of adjuvant CT in patients with pathological stage I NSCLC with high-risk features. Methods: The 2016 National Cancer Database was queried to identify patients with pathological stage I NSCLC (8th edition AJCC staging) diagnosed from 2010-2015 who received lobectomy/pneumonectomy with clear surgical margins. Patients were stratified into high risk (tumor size ≥2 cm with LVI and/or VPI) or low risk group. Multivariate Cox proportional hazards regression and propensity score matched Kaplan-Meier survival analysis were used to compare overall survival between those who received adjuvant CT and those who did not. Results: 34,556 patients were identified with 1114 (3.2%) receiving adjuvant CT. On multivariate Cox regression analysis, high risk tumors (hazard ratio [95% confidence interval] = 1.31 [1.25-1.38]) and lack of adjuvant chemotherapy (1.25 [1.09-1.44]) were associated with worse overall survival (OS). Additionally, male sex, age ≥ 60 years, higher comorbidity burden, lack of insurance, low facility volume, low median income, non-squamous histology were associated with worse OS. After propensity score matching, Kaplan-Meier survival analysis of the high risk subgroup (n = 2923) showed a significant difference in overall survival (OS) between those who received adjuvant CT (n = 1032, 5 year OS, 74.7%; 95% CI, 70.9%-78.0%) and those who did not (n = 1891, 5 year OS, 66.9%; CI, 63.9%-69.6%; p = 0.0002). In patients with no high risk factors for recurrence (n = 384), OS was not significantly different between the patients who received adjuvant CT (n = 78, 5 year OS, 75.8%; CI, 61.3%-85.5%) and those who did not receive adjuvant CT (n = 306, 5 year OS, 77.1%; CI, 70.0%-82.7%; p = 0.3). Conclusions: Our study showed better survival with adjuvant CT in patients with pathological stage I NSCLC who have tumor size greater than 2 cm, LVI and/or VPI.

Background: Primary cutaneous diffuse large B cell lymphoma-leg type (DLBCL-LT) is relatively uncommon lymphoma accounting for about 1 to 3 % of all cutaneous lymphoma and about 10-20% of primary cutaneous B cell lymphomas. There is data for survival outcomes in primary cutaneous B cell lymphoma in general population after introduction of rituximab but little is known about outcomes in DLBCL-LT patients in general population. The available data on survival in DLBCL-LT are from small case series and single institution studies. We conducted this study to evaluate the survival outcome in patients with PCDLBCL-LT after introduction of rituximab. Methods: We utilized SEER 18 database to select adult (18+ years) patients diagnosed from 2004 to 2014 with ICD-3-0 code of 9680/3 (diffuse large B cell lymphoma, not otherwise specified)with localized stage by summary stage 2000 with skin as the primary site of disease to identify patient with DLBCL-LT. Patients were divided into various cohorts based on age (<65 versus 65+ years), sex, race (Caucasians, African American and Others) and median household income of county of residence (< $50,00 versus ≥ $50,000). We utilized SEER*stat to calculate age adjusted incidence rate using 2000 US standard population. Kaplan Meyer curve was utilized to calculate 5-year overall survival. Cox proportional hazard model was used for multivariate analysis of factor associated with survival. Result: A total of 485 patients were identified with DLBCL-LT as the only or first primary cancer case. The median age at diagnosis was 70 years. Its incidence was found to be about 0.09 case per 100,000 population with incidence in males being about 1.6 times that of females. Majority of patient included in our study were males (52 %), older (59.4%), and Caucasians (85.6%). 5-year overall survival was found to be 59%. Older patients were found to have significantly worse 5-year overall survival compared to younger patients with HR of 7.9 with 95% CI of 4.9 to 12.7. There was no disparity in cancer outcome based on race, sex or median income of the county of residence. When we compared the outcome of patients diagnosed in earlier 5 years to later 6 years of the study period, the survival curves almost overlapped suggesting that there has been little if any improvement in outcome in recent years. Conclusion: Our study showed that about 60% of patients with DLBCL-LT live for 5 years in real world setting which is comparable to the outcome reported by other studies on outcome in primary cutaneous B cell lymphoma. Even though this is disease of elderly, as the median age at diagnosis is 70 years, the outcome in older population was found to be significantly worse compared to younger patients. Given this is relatively uncommon disease, we recommend large registry-based studies to try to improve outcome in these patient population. Disclosures No relevant conflicts of interest to declare.