Jajja, Salman Ayub

134 Background: Prostate cancer patients with visceral metastases at diagnosis are often treated using one-size-fits-all approach. For example, CHAARTED criteria treats any visceral metastasis as high volume. Therefore, we aimed to assess overall survival (OS) by site of visceral involvement in patients with metastatic PCa. Methods: Surveillance, Epidemiology, and End Results (SEER) database (2010-2021) was queried to obtain case listing data on metastatic PCa patients from 17 registries. The patients were categorized into three groups based on visceral metastasis at diagnosis: lung metastasis, liver metastasis, and others. Kaplan-Meier survival analysis was performed to estimate median OS with 95% confidence interval (CI). Log-rank test was conducted to assess if survival was significantly different among the groups. Cox proportional hazard regression analysis was performed to assess the magnitude of difference among the groups. P-value <0.05 established statistical significance. Results: This analysis included 21,007 metastatic PCa patients (lung: n=319; liver: n=123; brain: 107; bone: 20538). Metastatic PCa population with lung metastasis exhibited a median OS of 60 months (95% CI: 48-72) compared to 24 months (95% CI: 12-36) in population with liver metastases and to 36 months (95% CI: 36-36) in population with other metastases. The difference in OS among the groups was statistically significant (p < 0.001). Compared to population with lung metastasis, those with liver (HR: 1.91; 95% CI: 1.41-2.57) but not those with other metastases (1.16; 0.97-1.39) had significantly worse OS adjusting for Gleason score, PSA and race (Table). Conclusions: In metastatic PCa, patients with lung-only metastasis have significantly better overall survival and should be considered differently in prognostication to liver metastases. Consideration of site of visceral metastasis is critical for treatment intensification decisions. Limitations of this analysis include lack of accounting for confounding relationships at individual patient level. Variable Hazard Ratio (95% CI) Metastases (Lung) REFERENCE Metastases (Liver) 1.91 (1.41-2.57) Metastases (Other) 1.16 (0.97-1.39) Gleason score≥8 REFERENCE Gleason score <8 0.68 (0.64 -0.72) PSA - ng/dL 1.01 (1.00 – 1.01) Race (White) REFERENCE Race (Black) 0.99 (0.94 – 1.05) Race (Asian or Pacific Islander) 0.70 (0.64 – 0.76)

341 Background: Specialist clinicians (oncologists, urologists, radiation oncologists) read through free text reports of prostate MRI and biopsy reports to risk stratify patients during initial consultation of localized PCa. This laborious and time-consuming process is repeated for each day during the clinic. Herein, we propose a hybrid stratification framework utilizing a combination of large language model (LLM) and rule-based programming to automate this process. Methods: This retrospective study included patients with localized PCa (2004-2024) presenting to care at Mayo Clinic with at least one positive prostate biopsy and MRI report available. The state-of-the-art LLM – GPT4 – was utilized using a structured zeroshot prompt to extract relevant phenotypic variables (clinical T stage, Gleason patterns, grade group, extent of specimen/cores involved, prostate size/volume) from unstructured MRI and biopsy reports. Using a rule-based algorithm, the extracted variables and PSA at PCa diagnosis were used to categorize each patient into one of the six NCCN risk categories (very low [VL], low [L], favorable intermediate [FInt], unfavorable intermediate [UFInt], high risk (HR), very high risk [VHR]). Prompts were iteratively developed using ~5% of the total dataset and validated on ~10% of the dataset. Final performance was assessed against a held-out expert annotated test dataset using evaluation metrics (accuracy, F1-score, precision, recall). Additionally, manual annotation was conducted by two independent novice reviewers to compare machine annotated and novice-human annotated risk categorization. Results: A total of 397 patients were included in the evaluation. The median age at diagnosis was 64.8 (IQR: 59.7-68.5); majority of the men were White (n: 366; 92%) and non-Hispanic (n: 371; 94%). The most prevalent risk category was UFint (n: 151; 38%) followed by FInt (n: 80; 20.2%), HR (n: 76; 19.1%), VHR (n: 64; 16.1%), L (21: 5.3%), and VL (n: 5; 1.3%). Novice reviewers achieved an accuracy of 79%, precision of 80%, recall of 82% and a F1-score of 0.81. The hybrid stratification framework using GPT4 achieved an accuracy of 89%, precision of 89%, recall of 88% and a F1-score of 0.88. Among 43 errors by GPT4 risk stratification, majority of the errors were due to miscategorization of clinical T-stage (n: 21; 49%) followed by erroneous number of positive cores (n: 13; 30%). However, additional evaluation showed that GPT4 achieved numerically higher performance (accuracy: 90%; precision: 91%; recall: 90%; F1: 0.90) compared to novice reviewers (accuracy: 77%; precision: 89%; recall: 77%; F1: 0.82) for ascertaining clinical T-stage. Conclusions: Large language models exhibited superior performance than novice clinicians for risk stratification in patients with localized PCa. As the next step, we intend to validate our framework for automated risk classification in a prospective manner.

Introduction Prognostication in myelofibrosis (MF) currently relies on manual assessment of clinical, pathological, and laboratory variables, which is inefficient and error-prone. The potential of large language models (LLMs) and rule-based programming to automate data extraction and prognostication in these patients remains unexplored and merits investigation. Methods Patients with MF were identified from a retrospective chart review. Data regarding clinical (age, constitutional symptoms, transfusion requirements), laboratory (hemoglobin, circulating white blood cell, platelet, and blast counts), and pathological (grade of bone marrow fibrosis, chromosomal analyses, cytogenetics) features was extracted from the latest free-text clinical notes and pathology reports using the generative pretrained transformer-4 (GPT-4) model in a zero-shot setting with the prompting parameters set to generate deterministic responses. The patients were randomly sampled into development (15%) and test (85%) sets, with the documents of patients in the development set used for iterative prompt engineering. The finalized prompts were used for data extraction in the test set. Rule-based programming was used for post-processing of the extracted data and ascertaining prognosis via Dynamic International Prognostic Scoring System (DIPSS), DIPSS-plus (DIPSS+), and Mutation-enhanced International Prognostic Scoring System-70 (MIPSS-70). Automatically extracted data and prognostic categories were compared with the manually generated human annotations. Mean accuracy, precision, and recall with 95% confidence intervals (CI) were computed. Detailed error and sensitivity analyses were conducted to assess performance. Results The study included 100 patients (development: 15; test: 85) with 100 clinical notes (development: 15; test: 85) and 90 pathology reports (development: 13; test: 77). The mean age of these patients was 70 (SD: 10.6); 55 (55%) patients were men, and 93 (93%) patients were White. In the development set, the mean accuracy, precision, and recall for data extraction from clinical notes were 94% (95% CI: 90-98%), 92% (84-99%), and 99% (97-100%). For data extraction from pathology reports, the mean accuracy, precision, and recall were 96% (93-100%), 98% (92-100%), and 83% (63-100%). For prognostication, the mean accuracy, precision, and recall were 87% (79-94%), 80% (73-87%), and 100% using DIPSS (accuracy: 80%; precision: 73%; recall: 100%), DIPSS+ (accuracy: 93%; precision: 83%; recall: 100%) and MIPSS-70 (accuracy: 87%, precision: 83%; recall: 100%). In the test set, the mean accuracy, precision, and recall for data extraction from clinical notes were 96% (95% CI: 93-98%), 91% (87-96%), and 98% (97-100%). For data extraction from pathology reports, the mean accuracy, precision, and recall were 95% (92-98%), 93% (87-99%), and 90% (80-100%). For prognostication the mean accuracy, precision, and recall were 95% (94-96%), 89% (88-90%) and 99% (96-100%) using DIPSS (accuracy: 94%; precision: 89%; recall: 100%), DIPSS+ (accuracy: 95%; precision: 88%; recall: 95%) and MIPSS-70 (accuracy: 94%, precision: 90%; recall: 100%). Error analyses identified incorrect extractions by GPT-4 were related to the circulating blasts and constitutional symptoms variables from the clinical notes. Sensitivity analyses excluding the associated variables improved overall accuracy from 94% to 97% and 96% to 97% in the development and test sets, respectively. Similarly, incorrect extractions from the pathology reports were related to the CALR gene mutation variables. Sensitivity analyses excluding the associated variables improved the overall accuracy from 96% to 98% and 95% to 97% in the development and test sets, respectively. Conclusion LLMs and rule-based programming can accurately extract data from the electronic health records and prognosticate MF patients via DIPSS, DIPSS+, and MIPSS-70. The automated approach could significantly streamline clinical and research workflows, providing an efficient and less resource-intensive alternative to manual prognostication.