Shao, Yusra F.

e21598 Background: While large language models (LLMs) like GPT-4 (OpenAI) and Claude Opus (Anthropic) perform well on general medical exams, limited data exist on their application in oncology, particularly in multiple-choice question (MCQ) scenarios. This study benchmarks the accuracy and assesses the clinical reasoning of LLMs on cutaneous oncology MCQs from the American Society of Clinical Oncology (ASCO) question bank. Methods: Skin oncology MCQs from the ASCO question bank were tested using GPT-4 (OpenAI) and Claude Opus (Anthropic). Questions were entered into OpenAI and Anthropic API playgrounds without additional prompting, with identical settings for both models (Temperature = 0, Token = Max). Each question was tested three times per model to assess response consistency and precision. Initial responses were collected, and chain-of-thought (COT) prompting was applied to encourage stepwise reasoning mimicking an oncologist’s thought process. The percentage of revised answers after COT prompting was analyzed. Accuracy before and after COT prompting was compared. Incorrect responses were reviewed by board-certified oncologists specializing in skin cancer at National Cancer Institute (NCI)-designated centers. Each oncologist scored the responses on clarity, clinical reasoning, bias, and confidence in the models’ utility in clinical practice. Qualitative feedback was collected via free-text comments and analyzed descriptively. Results: Of 66 skin oncology MCQs, Claude correctly answered 55 questions (83% accuracy, 95% CI: 72%–91%), while ChatGPT correctly answered 53 (80% accuracy, 95% CI: 68%–89%). Both models performed comparably, with overlapping confidence intervals. A chi-square test showed no significant difference between their performances (p = 0.65). After COT prompting, both tools remained at their respective accuracy level. Thematic analysis of incorrect answers revealed that LLM responses frequently deviated from clinical guidelines and failed to reflect evidence-based oncology practice, often neglecting key trial data like DREAMSeq and frontline therapeutic strategies. The most frequent errors were Treatment Planning Errors, where recommendations misaligned with NCCN guidelines, and Interpretation Errors, involving misinterpretation of clinical trials or patient data. Anchoring bias emerged as a recurring issue, with models fixating on details (e.g., BRAF mutation status) at the expense of broader reasoning. Conclusions: This study demonstrates that while large language models like GPT-4 and Claude Opus show promise in the field of oncology, they still face significant challenges in clinical reasoning and adherence to evidence-based practices. Continued refinement and training on specialized medical data are essential before these models can be reliably integrated into clinical workflows.

Abstract Background: Uveal melanoma (UM) is a rare tumor of the eye which is biologically distinct from cutaneous melanoma (CM) and carries a poor prognosis once metastatic. We aimed to examine a multi-omics profile of UM as compared to CM. Methods: A total of 277 UM and 1,297 CM metastatic or primary samples were analyzed using next-generation sequencing (NextSeq, 592 genes and WES, NovaSeq, >700 genes) and whole transcriptome sequencing (WTS; NovaSeq) (Caris Life Sciences). Tumor mutational burden (TMB) was measured by totaling somatic mutations per tumor (TMB-high cutoff >10 mutations per MB). Immune cell fraction was calculated by QuantiSeq. T- cell inflamed score was calculated using gene expression data. Pathway gene enrichment analysis using WTS data was performed using GSEA. Biostatistical significance was determined using chi-square/Fisher-Exact or Wilcoxon rank sum test and adjusted for multiple comparisons. Results: For all cases (primary and metastatic), UM compared to CM had a higher rate of GNAQ/11 (93.8% vs. 2.1% p≤0.001), SF3B1 (20.4% vs 3% p≤0.001), and BAP1 (52.9% vs. 1.5% p≤0.001) mutations. UM also had higher levels of MYC (7% vs 0.9% p≤0.001), NCOA2 (3.1% vs 0.1% p≤0.001), and RUNX1T1 genomic amplifications (2.3% vs 0.2% p≤0.001). Assessing only samples of liver metastases, UM was shown to have higher abundance of M2 macrophages (P≤0.0001) and less abundance of M1 macrophages(P≤0.0001), monocytes(P≤0.0001), CD4 T-cells(P≤0.0001), and regulatory T-cells (p = 0.0003) as compared to CM. GSEA comparison between liver metastases of UM and CM samples showed G2M checkpoint (NES score, 1.60 p≤0.0001) and E2F targets (NES score, 1.58 p=0.002) to be higher in metastatic CM. When comparing T-cell inflamed, intermediate, and non-inflamed tumors from any site, CM or UM, the average Z scores for both G2M and E2F were highest for inflamed and lowest for non-inflamed, P≤0.0001. A similar trend of higher Z score was seen for TMB high vs TMB low tumors, p≤0.0001. Conclusions: UM and CM are characterized by significantly different genomic and transcriptomic alterations, and cellular composition of TME. Further understanding of these characteristics may lead to better treatment strategies for metastatic UM. Citation Format: Justin C. Moser, Yusra F. Shao, Joanne Xiu, Yasmine Baca, Takami Sato, Lauren A. Dalvin, Sourat Darabi, Michael Korn, Burton Eisenberg, Geoff Gibney, Evidio Domingo-Musibay, Gino In, Dave Hoon, Michael S. Gordon. Multi-omics and immune cell profiling of metastatic uveal melanomas [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2022; 2022 Apr 8-13. Philadelphia (PA): AACR; Cancer Res 2022;82(12_Suppl):Abstract nr 50.