Citations

e13637 Background: Large language models (LLMs) like GPT-4 (OpenAI) and Claude Opus (Anthropic) showed high accuracy in medical multiple-choice exams, but data on their oncology-specific clinical reasoning and performance is limited. This study evaluates their accuracy and clinical reasoning on breast oncology multiple-choice questions (MCQs) from the American Society of Clinical Oncology (ASCO) question bank. Methods: Using OpenAI and Anthropic Application Programming Interface (APIs), questions were tested without additional prompts under consistent settings (GPT-4 and Claude Opus; Temperature = 0, Tokens = Max). Each question was tested three times to assess precision. Then, Chain-of-thought (COT) prompting was applied to promote LLMs stepwise reasoning to increase their accuracy. Accuracy before and after COT prompting was compared. Incorrect responses were reviewed by board-certified medical oncologists speclized in breast cancer, who scored reasoning clarity, bias, and clinical relevance. Qualitative feedback was descriptively analyzed. Results: A total of 273 breast oncology MCQs were evaluated across the two LLMs. GPT-4 achieved an initial accuracy of 81.3% (222/273; 95% CI: 76.3%-85.5%), compared to Claude Opus, which achieved an accuracy of 79.5% (217/273; 95% CI: 74.3%-83.9%). The Chi-squared test for difference between the models before chain-of-thought (COT) prompting yielded a p-value of 0.59, indicating no statistically significant difference in accuracy between GPT-4 and Claude Opus prior to COT prompting. After COT prompting, the performance of the two models diverged significantly. GPT-4 saw a net decline in accuracy, decreasing by a net of 1 correct answer, resulting in an overall accuracy of 80.95% (221/273). In contrast, Claude Opus experienced a notable improvement, with a net of 19 additional correct answers, leading to an accuracy of 86.4% (236/273). A statistical analysis using a Chi-squared test revealed a difference in accuracy between the two models of 5.5% (p = 0.08), demonstrating that the improvement in accuracy for Claude Opus after COT prompting was borderline statistically significant compared to GPT-4. Thematic analysis of oncologists' feedback revealed that the most common reasons for incorrect answers were reliance on outdated guidelines, misinterpretation of clinical trial data, and failure to consider multidisciplinary or patient-specific approaches in clinical decision-making. Conclusions: Although AI models can achieve high scores on multiple-choice exams, they still require human supervision. These models rely on potentially outdated training data and lack the ability to individualize patient care or apply data from clinical trials. especially in unique or unconventional/non textbook scenarios.

Background: Thrombocytopenia is a frequently encountered laboratory abnormality and a common reason for hematology referrals. Workup for thrombocytopenia is not standardized and frequently does not follow an evidence-based algorithm. We conducted a systematic analysis to evaluate the laboratory testing and outcomes of patients evaluated for thrombocytopenia at hematology clinics in a tertiary referral center between 2013 and 2016. Patient and methods: We performed a comprehensive chart review for patients evaluated for thrombocytopenia during the study period. Patients were followed for 1 year from the initial hematology evaluation and assessed for the development of a hematologic malignancy, rheumatologic, or infectious diseases among other clinical outcomes. Results: We evaluated 472 patients with a median (range) age of 61 (17–94) years. The majority (63.8%) had mild thrombocytopenia. Within 1 year of follow-up, 14 patients (3.0%) were diagnosed with a hematologic malignancy. A higher likelihood of developing a hematologic malignancy was noted in patients with concurrent leukopenia (hazard ratio [HR] 9.97, 95% confidence interval [CI] 3.28–30.32, p < .01) and increasing age (HR per 10-year deciles 1.52, 95% CI 1.03–2.25, p = .03). In patients with asymptomatic isolated mild thrombocytopenia, laboratory testing did not reveal any significant positive findings and patients did not receive any new major diagnosis during the follow-up period. Conclusion: Our findings provide basis and call for development of an evidence-based algorithmic approach for evaluation of patients with thrombocytopenia, testing, and referrals. It also supports a conservative approach mainly driven by physical exam signs, symptoms, and other laboratory findings for patients with isolated mild thrombocytopenia.