He, Huan

Abstract Objective Data extraction from the published literature is the most laborious step in conducting living systematic reviews (LSRs). We aim to build a generalizable, automated data extraction workflow leveraging large language models (LLMs) that mimics the real-world 2-reviewer process. Materials and Methods A dataset of 10 trials (22 publications) from a published LSR was used, focusing on 23 variables related to trial, population, and outcomes data. The dataset was split into prompt development (n = 5) and held-out test sets (n = 17). GPT-4-turbo and Claude-3-Opus were used for data extraction. Responses from the 2 LLMs were considered concordant if they were the same for a given variable. The discordant responses from each LLM were provided to the other LLM for cross-critique. Accuracy, ie, the total number of correct responses divided by the total number of responses, was computed to assess performance. Results In the prompt development set, 110 (96%) responses were concordant, achieving an accuracy of 0.99 against the gold standard. In the test set, 342 (87%) responses were concordant. The accuracy of the concordant responses was 0.94. The accuracy of the discordant responses was 0.41 for GPT-4-turbo and 0.50 for Claude-3-Opus. Of the 49 discordant responses, 25 (51%) became concordant after cross-critique, increasing accuracy to 0.76. Discussion Concordant responses by the LLMs are likely to be accurate. In instances of discordant responses, cross-critique can further increase the accuracy. Conclusion Large language models, when simulated in a collaborative, 2-reviewer workflow, can extract data with reasonable performance, enabling truly “living” systematic reviews.

Abstract Background Error analysis plays a crucial role in clinical concept extraction, a fundamental subtask within clinical natural language processing (NLP). The process typically involves a manual review of error types, such as contextual and linguistic factors contributing to their occurrence, and the identification of underlying causes to refine the NLP model and improve its performance. Conducting error analysis can be complex, requiring a combination of NLP expertise and domain-specific knowledge. Due to the high heterogeneity of electronic health record (EHR) settings across different institutions, challenges may arise when attempting to standardize and reproduce the error analysis process. Objectives This study aims to facilitate a collaborative effort to establish common definitions and taxonomies for capturing diverse error types, fostering community consensus on error analysis for clinical concept extraction tasks. Materials and Methods We iteratively developed and evaluated an error taxonomy based on existing literature, standards, real-world data, multisite case evaluations, and community feedback. The finalized taxonomy was released in both .dtd and .owl formats at the Open Health Natural Language Processing Consortium. The taxonomy is compatible with several different open-source annotation tools, including MAE, Brat, and MedTator. Results The resulting error taxonomy comprises 43 distinct error classes, organized into 6 error dimensions and 4 properties, including model type (symbolic and statistical machine learning), evaluation subject (model and human), evaluation level (patient, document, sentence, and concept), and annotation examples. Internal and external evaluations revealed strong variations in error types across methodological approaches, tasks, and EHR settings. Key points emerged from community feedback, including the need to enhancing clarity, generalizability, and usability of the taxonomy, along with dissemination strategies. Conclusion The proposed taxonomy can facilitate the acceleration and standardization of the error analysis process in multi-site settings, thus improving the provenance, interpretability, and portability of NLP models. Future researchers could explore the potential direction of developing automated or semi-automated methods to assist in the classification and standardization of error analysis.

Abstract Despite recent methodology advancements in clinical natural language processing (NLP), the adoption of clinical NLP models within the translational research community remains hindered by process heterogeneity and human factor variations. Concurrently, these factors also dramatically increase the difficulty in developing NLP models in multi-site settings, which is necessary for algorithm robustness and generalizability. Here, we reported on our experience developing an NLP solution for Coronavirus Disease 2019 (COVID-19) signs and symptom extraction in an open NLP framework from a subset of sites participating in the National COVID Cohort (N3C). We then empirically highlight the benefits of multi-site data for both symbolic and statistical methods, as well as highlight the need for federated annotation and evaluation to resolve several pitfalls encountered in the course of these efforts.

5088 Background: Previous evidence suggests that there may be no additional benefit of triplet therapy in low volume disease based on limited data. Therefore, we investigated the efficacy of triplet therapy as compared to docetaxel (D) and androgen pathway inhibitor (API) doublets by volume of disease using the most up to date results from the ARASENS trial. Methods: Phase III randomized controlled trials (RCTs) assessing treatment intensification with API, and/or D in patients with mCSPC were included. Precomputed hazard ratios (HR) with 95% confidence intervals (CI) for OS were pooled using an inverse-variance approach. A DerSimonian-Laird random-effects meta-analysis was conducted to assess the efficacy of triplet therapy compared to D doublet therapy. P-value of interaction was computed to assess difference between high (HV) and low volume (LV) disease subgroups (Pint <0.1 - statistical significance). Additionally mixed treatment comparisons were computed using network meta-analysis (NMA) to assess the comparative effectiveness of triplet therapy compared to API doublets by volume of disease. Results: Pairwise meta-analysis included a total of 3 RCTs (ARASENS, PEACE-1, ENZAMET) with 2518 patients (HV: 1820; LV: 698) as shown. In patients with HV disease, 385 (43%) and 484 (53%) deaths were observed with triplet therapy, and D doublet, respectively. Triplet therapy significantly improved OS as compared to D doublet in HV (HR: 0.73; 95% CI: 0.64-0.84). In patients with LV disease, 73 (20%) and 94 (28%) deaths were observed with triplet therapy, and D doublet, respectively. Triplet therapy significantly improved OS as compared to D doublet in LV (HR: 0.71; 95% CI: 0.52-0.97). There was no statistically significant interaction by volume of disease for triplet therapy vs. D doublet (Pint: 0.86). NMA including 10 clinical trials and over 11500 patients updated as of 13th Feb 2023 showed that in HV mCSPC, triplet therapy was ranked as potentially the most efficacious treatment option and may improve OS compared to API doublet therapy (HR: 0.83; 95% CI: 0.66-1.03). In LV mCSPC, API doublet therapy was ranked as potentially the most efficacious treatment followed by triplet therapy. There was no significant difference between triplet therapy and API doublet therapy (HR: 1.12; 95% CI: 0.74-1.12). Conclusions: These results underscore that triplet therapy may be preferred in mCSPC patients with HV disease whereas API doublet therapy may be preferred in LV disease. [Table: see text]

695 Background: Recent results from COSMIC 313 trial and updated findings from other contemporary trials have added to the existing body of evidence for 1L mRCC. Therefore, we present the most up-to-date results from our living, interactive systematic review (LISR) to facilitate the choice of optimal therapy in 1L mRCC setting. Methods: This LISR and network meta-analysis is maintained using a novel living evidence synthesis (LIvE) framework. The framework facilitates the identification of new or updated studies using an automated ‘living’ search which is followed by screening and extraction in a semi-automated fashion within a machine learning assisted graphical user interface. Relevant data is parsed and analyzed to compute mixed treatment comparisons and certainty of evidence is adjudicated using a rule-based algorithm. Detailed methods have been published before (PMID: 33824031). Results: As of October 1st 2022, this LISR includes 16 clinical trials (30 references). Mixed treatment comparisons showed statistically significant PFS benefit with lenvatinib-pembrolizumab (LenPem; rank 1) when compared to cabozantinib-nivolumab (CabNivo; rank 3; hazard ratio [HR]: 0.69, 95% confidence interval: 0.52-0.93), CaboNivo-ipilimumab (CaboNivoIpi; rank 4; HR: 0.62; 0.43-0.89), avelumab-axitinib (AveAxi; rank 5; HR: 0.56; 0.43-0.75), PemAxi (rank 6; HR: 0.55; 0.42-0.72), atezolizumab-bevacizumab (AteBev; rank 7; HR: 0.46; 0.35-0.59), and NivoIpi (rank 8; HR: 0.45; 0.35-0.59). No significant differences were observed between CaboNivoIpi and other combination therapies for PFS improvement. Similarly, for OS, LenPem (rank 1), PemAxi (rank 2) and CaboNivo (rank 3) were ranked as potentially more efficacious treatment options than other counterparts. However, no statistically significant difference was observed for OS benefit with mixed treatment comparisons. The odds of achieving an ORR were higher with LenPem (rank 1; odds ratio [OR]: 2.44; 1.43-4.17), and CaboNivo (rank 2; OR: 1.77; 1.04-3.03) when compared to CaboNivoIpi (rank 6). NivoIpi (rank 1), LenPem (rank 2), CaboNivoIpi (rank 3) were ranked potentially as more efficacious treatments for achieving a CR. In terms of grade 3 or higher treatment related adverse events, NivoIpi (rank 3) was ranked as potentially the safest option among other combinations while CaboNivoIpi (rank 12), LenPem (rank 11), and CaboNivo (rank 10) were more likely to increase the risk of toxicity compared to most other treatments. Conclusions: Triplet combination of CaboNivoIpi may not provide additional survival benefit compared to other immunotherapy combinations and may increase toxicity. Patient-level considerations such as treatment toxicity, and quality of life should be factored in when opting intensified 1L therapies in mRCC patients.

694 Background: The treatment landscape of localized renal cell carcinoma (RCC) is rapidly evolving. Recent trials have demonstrated contrasting efficacy with adjuvant immunotherapy. Therefore, we sought to assess the mixed treatment comparisons among different adjuvant treatment options using updated data from trials and quantified the absolute effect with these treatments stratified by risk categories. Methods: This living network meta-analysis was conducted using the living interactive evidence (LIvE) synthesis framework. We used stratified baseline risks of disease progression from observational data and at 5, 10, 15 years from the Leibovich risk stratification system (based on risk scores ranging from 0 to ≥15). Corresponding intervention risks were then approximated using relative effect estimates (from NMA) and baseline risks. The difference between CIRs and baseline risks were calculated to present absolute risk differences in each risk category. Results: This NMA included eight RCTs with 8480 participants and seven unique treatment options. Pembrolizumab (pembro; rank 1) was associated with improved disease-free survival (DFS) when compared to atezolizumab (atezo; rank 6; hazard ratio: 0.68; 0.49;0.93), and nivolumab-ipilimumab (nivoipi; rank 5; 0.68; 0.48-0.97). However, no statistically significant difference was observed between pembro and atezo for overall survival (0.53; 0.28-1.01). Survival data for nivoIpi was not reported. No new statistically significant differences were observed since last update. The absolute benefit of atezo and nivoipi was minimal with higher T and N patients (Table). The results were similar with increasing Leibovich risk scores. Conclusions: Current evidence favors the use of a risk adapted approach when offering adjuvant immunotherapy. Adjuvant pembrolizumab remains a preferred treatment in patients with RCC who underwent nephrectomy. [Table: see text]

193 Background: We maintain a network meta-analysis for the treatment of mCSPC using a living interactive evidence (LIvE) synthesis framework. Recently, the STAMPEDE trial presented evidence regarding a new triplet regimen (enzalutamide-abiraterone-androgen deprivation therapy [E+AAP+ADT]). Hence, in this report we provide the most-up-to-date evidence to inform the choice of optimal therapy in mCSPC patients. Methods: Phase III randomized controlled trials (RCTs) assessing treatment intensification with androgen pathway inhibitors (API), docetaxel (D), or both in patients with mCSPC were included. Mixed treatment comparisons for overall-survival (OS) and progression-free survival (PFS) were made across randomized comparisons using a frequentist network-meta-analysis. Additionally, subgroup data from three trials (ENZAMET, TITAN, ARCHES) by the receipt of docetaxel were abstracted to evaluate the comparative effectiveness of different triplet regimens. Treatment effects were expressed as hazard ratio (HR) and 95% confidence intervals (CI). Relative treatment rankings were evaluated using P-scores and assessed in congruency with the pairwise estimates. Results: This report includes 10 clinical trials (30 references) and over 11500 patients updated as of 1st October 2022. Using randomized data in overall patient population, AAP+D+ADT (rank 1; HR: 0.50; 95% CI: 0.35-0.72) and E+AAP+ADT (rank 4; HR: 0.76; 0.62-0.93) significantly improved PFS compared to D+ADT (rank 7) but not when compared to E+ADT (rank 2) and APA+ADT (rank 3). Similarly, darolutamide (DARO)+D+ADT (rank 1; HR: 0.68; 0.57-0.81), AAP+D+ADT (rank 2; HR: 0.75; 0.59-0.95) significantly improved OS when compared to D+ADT (rank 7). Efficacy by volume of disease and timing of metastatic presentation was consistent with prior update. Using subgroup data by the receipt of docetaxel, a total of five triplet regimens (DARO+D+ADT, AAP+D+ADT, E+D+ADT, apalutamide (APA)+D+ADT, and E+AAP+ADT) were evaluable for comparisons. E+D+ADT (rank 3) improved PFS compared to D+ADT (rank 9; HR: 0.52; 0.41-0.66) and E+AAP+ADT (rank 6; HR: 0.70; 0.52-0.94). Compared to AAP+ADT (rank 7), E+D+ADT (HR: 0.64; 0.49-0.84) and AAP+D+ADT (rank 2; 0.62; 0.42-0.91) significant improved PFS. However, no significant differences were observed among E+D+ADT, AAP+D+ADT, and APA+D+ADT. Similarly, DARO+D+ADT (rank 1) improved OS compared to APA+D+ADT (HR: 0.47; 0.24-0.93). Conclusions: Current evidence favors the use of emerging triplet regimens over docetaxel doublet but not API doublets in overall population. Among different triplets, E+AAP+ADT may not delay progression compared to docetaxel triplets. Hence, access to treatment, toxicity, disease volume and timing of metastases should be carefully considered in mCSPC patients.

Importance The effectiveness of triplet therapy compared with androgen pathway inhibitor (API) doublets in a heterogeneous patient population with metastatic castration-sensitive prostate cancer (mCSPC) is unknown. Objective To assess the comparative effectiveness of contemporary systemic treatment options for patients with mCSPC across clinically relevant subgroups. Data Sources For this systematic review and meta-analysis, Ovid MEDLINE and Embase were searched from each database’s inception (MEDLINE, 1946; Embase, 1974) through June 16, 2021. Subsequently, a “living” auto search was created with weekly updates to identify new evidence as it became available. Study Selection Phase 3 randomized clinical trials (RCTs) assessing first-line treatment options for mCSPC. Data Extraction and Synthesis Two independent reviewers extracted data from eligible RCTs. The comparative effectiveness of different treatment options was assessed with a fixed-effect network meta-analysis. Data were analyzed on July 10, 2022. Main Outcomes and Measures Outcomes of interest included overall survival (OS), progression-free survival (PFS), grade 3 or higher adverse events, and health-related quality of life. Results This report included 10 RCTs with 11 043 patients and 9 unique treatment groups. Median ages of the included population ranged from 63 to 70 years. Current evidence for the overall population suggests that the darolutamide (DARO) triplet (DARO + docetaxel [D] + androgen deprivation therapy [ADT]; hazard ratio [HR], 0.68; 95% CI, 0.57-0.81), as well as the abiraterone (AAP) triplet (AAP + D + ADT; HR, 0.75; 95% CI, 0.59-0.95), are associated with improved OS compared with D doublet (D + ADT) but not compared with API doublets. Among patients with high-volume disease, AAP + D + ADT may improve OS compared with D + ADT (HR, 0.72; 95% CI, 0.55-0.95) but not compared with AAP + ADT, enzalutamide (E) + ADT, and apalutamide (APA) + ADT. For patients with low-volume disease, AAP + D + ADT may not improve OS compared with APA + ADT, AAP + ADT, E + ADT, and D + ADT. Conclusions and Relevance The potential benefit observed with triplet therapy must be interpreted with careful accounting for the volume of disease and the choice of doublet comparisons used in the clinical trials. These findings suggest an equipoise to how triplet regimens compare with API doublet combinations and provide direction for future clinical trials.

AbstractAn increasing number of studies have reported using natural language processing (NLP) to assist observational research by extracting clinical information from electronic health records (EHRs). Currently, no standardized reporting guidelines for NLP‐assisted observational studies exist. The absence of detailed reporting guidelines may create ambiguity in the use of NLP‐derived content, knowledge gaps in the current research reporting practices, and reproducibility challenges. To address these issues, we conducted a scoping review of NLP‐assisted observational clinical studies and examined their reporting practices, focusing on NLP methodology and evaluation. Through our investigation, we discovered a high variation regarding the reporting practices, such as inconsistent use of references for measurement studies, variation in the reporting location (reference, appendix, and manuscript), and different granularity of NLP methodology and evaluation details. To promote the wide adoption and utilization of NLP solutions in clinical research, we outline several perspectives that align with the six principles released by the World Health Organization (WHO) that guide the ethical use of artificial intelligence for health.