Jain, Atul

PURPOSE To evaluate open-source large language models (LLMs) for extracting cancer-specific phenotypic data, benchmark their performance against GPT4 models, and assess the impact of fine-tuning with training data sizes. METHODS Open-source LLMs (Mistral, LLaMa, MAMBA, BioMistral) were evaluated in zero-/one-shot and fine-tuned setups against GPT4-turbo/GPT4o to extract the cancer presence, progression, response, and metastatic sites from radiology impressions of patients with solid tumors treated at Dana-Farber Cancer Institute. Performance metrics (accuracy, precision, recall, F1-score) were computed. McNemar's odds ratio (OR), measuring which model is more likely to be correct when they disagree, was computed with 95% CI. Statistical significance was assessed using the alpha of .000139. RESULTS This study included 2,623 patients (25,273 radiology impressions). In zero-/one-shot settings, GPT4-turbo/GPT4o outperformed open-source LLMs. However, fine-tuned open-source LLMs achieved higher F1-scores than GPT4 models. Compared with the best-performing GPT4 model, fine-tuned Mistral0.2-7.3B (OR, 0.27 [95% CI, 0.20 to 0.36]; P < .00001), Mistral0.3-7.3B (OR, 0.26 [95% CI, 0.19 to 0.36]; P < .00001), LLaMa2-6.7B (OR, 0.30 [95% CI, 0.22 to 0.40]; P < .00001), LLaMa3.1-8B (OR, 0.37 [95% CI, 0.28 to 0.48]; P < .00001), and MAMBA-2.8B (OR, 0.32 [95% CI, 0.24 to 0.42]; P < .00001) showed significantly better performance in ascertaining disease progression. Performance was consistently better for inferring overall response, any evidence of cancer, and sites of metastases, with no significant differences among fine-tuned open-source LLMs. Fine-tuning gains plateaued at 25% of training data (5,718 impressions) and remained comparable at 5% (1,144 impressions). CONCLUSION Open-source LLMs, when fine-tuned using labeled data, can effectively automate the ascertainment of key radiophenotypic variables using only the impression section of radiology reports, without the full report text. Their consistent performance in small training sets suggests that these models may provide a scalable approach for phenotypic characterization of patients with cancer in real-world clinical settings.

PURPOSE: Financial hardship (FH) in cancer care is a growing challenge for patients, their caregivers, and health care providers with impact on morbidity and mortality. In this study, we report on a standardized approach to describe the prevalence and predictors for FH as part of routine clinical workflow. We also report on the association of FH with survival in our cancer patient population. METHODS: This study includes patients who completed a FH screen at least once between 2018 and 2020. Demographics, disease state, and mortality data were extracted from the medical records. Multivariable logistic regression models were used to examine association of sociodemographic and disease variables with FH. By using propensity score weighting to account for differences in demographic and clinical factors between patients with and without FH, we then fit Cox proportional hazards models to examine the relationship between FH and survival. RESULTS: The study cohort included 31,154 patients. FH was reported by 14% (n = 4,250) of the patients. A significantly higher likelihood of having FH ( P < .001 for all) was reported by racial/ethnic minority patients; those who were unemployed/disabled, single, or divorced; patients from disadvantaged neighborhoods; and those who were self-pay or had government insurance. Older age, being retired, and living farther from the cancer center were associated with significantly less likelihood of endorsing FH. Patients who endorsed FH had a lower survival (hazard ratio for mortality 1.46). CONCLUSION: Our study identified key groups more likely to report FH in a relatively affluent population at a large cancer center and showed an adverse association between FH and survival. Further research is needed to develop clinical care pathways for patients at high risk for worse financial and clinical outcomes.