Rawal, Samarth

Artificial intelligence (AI) is now applied across oncology, including imaging, pathology, therapy selection, toxicity assessment, and survival modeling, and large language models (LLMs) have been adopted particularly quickly. Reported physician use of AI roughly doubled between 2023 and 2026. The frameworks used to evaluate these tools have not advanced at the same rate. Many are available, but each was designed for a single stage of the AI lifecycle, and none directly answers the question most relevant to a treating clinician: whether a given tool is appropriate for the individual patient under care. We reviewed the frameworks in current use, including TRIPOD+AI, PROBAST+AI, CLAIM, SPIRIT-AI, CONSORT-AI, DECIDE-AI, MINIMAR, and CREMLS, together with the oncology-specific ESMO EBAI and the newer LLM-specific guidance (TRIPOD-LLM, MI-CLEAR-LLM, CHART, and ESMO ELCAP). Each addresses part of the evaluation problem, but none is sufficient on its own for oncology, where the standard of care changes rapidly, assays drift, biomarker-defined subgroups are small, and a model validated in one period may perform poorly in the next. LLMs introduce additional challenges, including sensitivity to prompting, undisclosed vendor updates, behavioral drift, and hallucination rates approaching 50% on clinical quality-assurance tasks. The predictive-model frameworks were not designed to capture these failure modes. The central argument of this review is that the frameworks the field requires already exist; what is missing is their mandatory adoption. We propose that journals and regulators move from recommending these frameworks to requiring their use, and we outline a cancer-aware evaluation pathway together with the specific responsibilities of authors, reviewers, journals, and regulators in implementing it.

Abstract Background The ability of large language models (LLMs) to work collaboratively and screen studies in a systematic review (SR) is under-explored. Hence, we aimed to evaluate the effectiveness of LLMs in automating the process of screening in systematic reviews. Methods This is an observational study which included labeled data (title and abstracts) for five SRs. Originally, two reviewers screened the citations independently for eligibility. A third reviewer cross-checked each citation for quality assurance. GPT-4, Claude-3-Sonnet, and Gemini-Pro-1.0 were used using zero-shot chain-of-thought prompting. Collaborative approaches included (i): conflict resolution using benefit of the doubt, (ii) majority voting using an independent third LLM and (iii) conflict resolution using an informed third LLM. Performance was assessed using accuracy, precision for exclusion, and recall for inclusion. Work saved over samples (WSS) was computed to estimate the reduction in manual human effort. Results A total of 11300 articles were included in this study. The individual models, GPT-4, Claude-3-Sonnet, and Gemini-Pro-1.0 exhibited a high precision for exclusion, achieving 99.7%, 99.7%, and 99.2% and high recall for inclusion achieving 95.5%, 96.6% and 85.7%, respectively. However, the collaborative approach utilizing the two best-performing models (GPT-4 and Claude-3S) achieved an average precision of 99.9% and a recall of 98.5% (across all collaborative approaches). Furthermore, the proposed collaborative approach resulted in an average WSS of 63.5%, compared to the average WSS of 45.2% for individual models. Conversational LLM interactions showed a consistent pattern of results. Limitations This study was limited due to reliance on proprietary models, and evaluation on oncology datasets. Conclusion Evidence shows that collaborative LLMs enable efficient, high-performing screening in systematic reviews, supporting continuous evidence updates. Primary funding source NIH (U24CA265879-01-1) and Carolyn-Ann-Kennedy-Bacon Fund.

2054 Background: Systematic reviews that summarize the toxicity of Immune checkpoint inhibitors (ICIs) become outdated very soon after publication. Therefore, we reported results of a toxicity meta-analysis at 2019 ASCO meeting and informed the intent to create a living systematic review (LSR). LSRs combine human and machine effort and support rapid evidence synthesis and living clinical practice guidelines. Now, we report our experience maintaining a LSR on toxicity of ICIs. Methods: Steps include quarterly literature searches to identify new clinical trials reporting ICI-associated adverse events (AEs), AI-enabled screening of new citations which meet the inclusion criteria, automated cumulative meta-analysis and an online reporting platform. Standard data formats and protocols were designed for inputting text, tables and graphics. Software was written to interpret these data and output the information in the appropriate format, such as a forest plot and summary tables. Finally, a dynamic interface that enables user inputs and displays the associated output was designed. Results: The LSR is continuously updated incorporating toxicity data from new clinical trials as it becomes available. We have screened 8000 relevant citations and summarized the odds of Grade 3 or higher AEs and AEs of special interest in patient receiving ICIs. The results are updated on quarterly basis and are available online. The results are updated on quarterly basis and will be available on a website at the time of publication. Prototype with dummy data is available at this link . This interface can also be manipulated via user input to organize and sort data tables and forest plots by type of cancer, name or mechanism (PD-1 or PD-L1) of ICI agent, single agent or combination, type of control arm, line of treatment and several other clinically relevant filters. For example, a user can instantaneously generate a meta-analysis summarizing the risk of colitis or pneumonitis in metastatic lung cancer trials with pembrolizmuab. Conclusions: This LSR engine can prospectively synthesize toxicity data from ICI trials in an efficient manner providing accurate and timely information for advanced clinical decision support at point-of-care. Efforts are ongoing to improve efficiency of screening, improve AI-enabled processes for automated screening and data abstraction, and test across multiple clinical questions.

80 Background: Several previous systematic reviews and meta-analyses have attempted to summarize toxicity of Immune Checkpoint inhibitors (ICIs). However, very soon after each one of these reviews has been published, it became outdated. ICIs are currently used in 14 different cancers and data is rapidly evolving from new clinical trials. A living Systematic review, which is defined as a systematic review that is continually updated to incorporate relevant new evidence as it becomes available, is necessary in this situations. Methods: In order to dynamically display content -- including textual data, tables, charts, and graphs – standard data formats and protocols were designed for various components of the systematic reviews carried out by researchers. Given these standardized formats, software was written to interpret this data and output the information in the appropriate format, such as a forest plot or data table. Finally, a dynamic interface that enables user inputs (such as which aspect of the meta-analysis to view) and displays the associated output was designed. The overall living Systematic Review infrastructure includes monthly literature searches, cumulative meta-analysis, and an online reporting platform. Results: We have developed a Living Systematic Review engine that can take in standardized data and reflect this data in an interactive, intuitive manner in the form of text, tables, and plots. This interface can also be manipulated via user input to organize and sort data tables or omit/highlight specific pieces of data. Conclusions: We have initiated the first living systematic review in immune-oncology that will be continuously updated, incorporating relevant new evidence as it becomes available, and will provide accurate and up to date toxicity estimates to support clinical decision making. The Living Systematic Review interface allows users to easily and intuitively access the most recent available clinical evidence on toxicity of ICIs.