Smithy, James W.

e21598 Background: While large language models (LLMs) like GPT-4 (OpenAI) and Claude Opus (Anthropic) perform well on general medical exams, limited data exist on their application in oncology, particularly in multiple-choice question (MCQ) scenarios. This study benchmarks the accuracy and assesses the clinical reasoning of LLMs on cutaneous oncology MCQs from the American Society of Clinical Oncology (ASCO) question bank. Methods: Skin oncology MCQs from the ASCO question bank were tested using GPT-4 (OpenAI) and Claude Opus (Anthropic). Questions were entered into OpenAI and Anthropic API playgrounds without additional prompting, with identical settings for both models (Temperature = 0, Token = Max). Each question was tested three times per model to assess response consistency and precision. Initial responses were collected, and chain-of-thought (COT) prompting was applied to encourage stepwise reasoning mimicking an oncologist’s thought process. The percentage of revised answers after COT prompting was analyzed. Accuracy before and after COT prompting was compared. Incorrect responses were reviewed by board-certified oncologists specializing in skin cancer at National Cancer Institute (NCI)-designated centers. Each oncologist scored the responses on clarity, clinical reasoning, bias, and confidence in the models’ utility in clinical practice. Qualitative feedback was collected via free-text comments and analyzed descriptively. Results: Of 66 skin oncology MCQs, Claude correctly answered 55 questions (83% accuracy, 95% CI: 72%–91%), while ChatGPT correctly answered 53 (80% accuracy, 95% CI: 68%–89%). Both models performed comparably, with overlapping confidence intervals. A chi-square test showed no significant difference between their performances (p = 0.65). After COT prompting, both tools remained at their respective accuracy level. Thematic analysis of incorrect answers revealed that LLM responses frequently deviated from clinical guidelines and failed to reflect evidence-based oncology practice, often neglecting key trial data like DREAMSeq and frontline therapeutic strategies. The most frequent errors were Treatment Planning Errors, where recommendations misaligned with NCCN guidelines, and Interpretation Errors, involving misinterpretation of clinical trials or patient data. Anchoring bias emerged as a recurring issue, with models fixating on details (e.g., BRAF mutation status) at the expense of broader reasoning. Conclusions: This study demonstrates that while large language models like GPT-4 and Claude Opus show promise in the field of oncology, they still face significant challenges in clinical reasoning and adherence to evidence-based practices. Continued refinement and training on specialized medical data are essential before these models can be reliably integrated into clinical workflows.

9517 Background: Although PD-1/L1 axis directed therapies induce durable responses in some mm patients (pts), biomarkers of response remain elusive. We hypothesized that quantifying key immune suppression mechanisms within the tumor microenvironment would provide superior predictors of response to anti-PD-1 compared with single marker assessment. Methods: Pre-treatment tumor biopsies from 124 mm pts treated with anti-PD-1 at 7 academic centers were fluorescently stained with multiple immune markers in discovery (n = 24) and validation (n = 100) cohorts. Selected biomarker signatures, PD-1/PD-L1 interaction score (proportion of PD-1+ cells co-localized with PD-L1) and IDO1/HLA-DR co-expression were evaluated for anti-PD-1 treatment response and survival. Slides were imaged using Vectra; biomarker positive cells and their co-localization were objectively quantified in pathologist-selected regions using novel Automated Quantitative Analysis (AQUA) algorithms. Results: In the discovery cohort, high levels of PD-1/PD-L1 interaction score and/or IDO1/HLA-DR coexpression was strongly positively associated with response to anti-PD-1 (p = 0.0005). In contrast, other individual biomarkers (PD-1, PD-L1, CD8) were not associated with response or survival (p > 0.10). This finding was replicated in the validation cohort: pts with high PD-1/PD-L1 and/or IDO1/HLA-DR were more likely to respond to treatment (p = 0.009). These pts also experienced a three-fold increase in progression free survival (hazards ratio (HR) = 0.33; p = 0.003) and overall survival (HR = 0.34; p = 0.004). Multivariate analyses revealed that these findings were independent of BRAF mutation, stage, LDH and prior therapy. In the combined cohort (n = 124), 80% of responding pts had higher levels of PD-1/PD-L1 interaction scores and/or IDO1/HLA-DR. In contrast, PD-L1 expression alone (≥1% or ≥50%) was not predictive of PFS or OS (p > 0.1). Conclusions: This novel multiplexed method profiling key tumor-immune suppression pathways identified mm pts likely to respond to anti-PD-1 therapy. This method could help stratify patients for PD-1 monotherapy and be useful in guiding future clinical trials.

Abstract Purpose: PD-1/L1 axis–directed therapies produce clinical responses in a subset of patients; therefore, biomarkers of response are needed. We hypothesized that quantifying key immunosuppression mechanisms within the tumor microenvironment by multiparameter algorithms would identify strong predictors of anti–PD-1 response. Experimental Design: Pretreatment tumor biopsies from 166 patients treated with anti–PD-1 across 10 academic cancer centers were fluorescently stained with multiple markers in discovery (n = 24) and validation (n = 142) cohorts. Biomarker-positive cells and their colocalization were spatially profiled in pathologist-selected tumor regions using novel Automated Quantitative Analysis algorithms. Selected biomarker signatures, PD-1/PD-L1 interaction score, and IDO-1/HLA-DR coexpression were evaluated for anti–PD-1 treatment outcomes. Results: In the discovery cohort, PD-1/PD-L1 interaction score and/or IDO-1/HLA-DR coexpression was strongly associated with anti–PD-1 response (P = 0.0005). In contrast, individual biomarkers (PD-1, PD-L1, IDO-1, HLA-DR) were not associated with response or survival. This finding was replicated in an independent validation cohort: patients with high PD-1/PD-L1 and/or IDO-1/HLA-DR were more likely to respond (P = 0.0096). These patients also experienced significantly improved progression-free survival (HR = 0.36; P = 0.0004) and overall survival (HR = 0.39; P = 0.0011). In the combined cohort, 80% of patients exhibiting higher levels of PD-1/PD-L1 interaction scores and IDO-1/HLA-DR responded to PD-1 blockers (P = 0.000004). In contrast, PD-L1 expression was not predictive of survival. Conclusions: Quantitative spatial profiling of key tumor-immune suppression pathways by novel digital pathology algorithms could help more reliably select melanoma patients for PD-1 monotherapy. Clin Cancer Res; 24(21); 5250–60. ©2018 AACR.