Tong, Jiayi

ABSTRACT Introduction Publication bias (PB) threatens the validity of meta‐analyses by distorting effect size estimates, potentially leading to misleading conclusions. With advanced pattern recognition and multimodal capabilities, large language models (LLMs) may be able to evaluate PB and make the systematic review process more efficient. Methods We evaluated the ability of two state‐of‐the‐art multimodal LLMs, GPT‐4o and Llama 3.2 Vision, to detect PB using funnel plots alone and in combination with quantitative inputs. We simulated meta‐analyses under varying conditions, including the absence of PB, different levels of presence of PB, varying total number of studies within a meta‐analysis, and differing degrees of between‐study heterogeneity. Results Neither GPT‐4o nor Llama 3.2 Vision consistently detected the presence of PB across various settings. Under no‐publication‐bias conditions, GPT‐4o achieved a higher specificity outperforming Llama 3.2 Vision, with the difference most shown in the meta‐analyses with 20 or more studies. The inclusion of quantitative inputs alongside funnel plots did not significantly improve performance. Additionally, between‐study heterogeneity and patterns of non‐reported studies had minimal impact on the models’ assessments. Conclusions The ability of LLMs to detect PB without fine‐tuning is limited at the present time. This study highlights the need for specialized model adaptation before LLMs can be effectively integrated into meta‐analysis workflows. Future research can focus on targeted refinements to enhance LLM performance and utility in evidence synthesis.

AbstractPurposeSevere adverse events (AEs), such as Guillain‐Barré syndrome (GBS) occur rarely after influenza vaccination. We identify highly associated AEs with GBS and develop prediction models for GBS using the US Vaccine Adverse Event Reporting System (VAERS) reports following trivalent influenza vaccination (FLU3).MethodsThis study analyzed 80 059 reports from the US VAERS between 1990 and 2017. Several AEs were identified as highly associated with GBS and were used to develop the prediction model. Some common and mild AEs that were suspected to be underreported when GBS occurred simultaneously were removed from the final model. The analyses were validated using European influenza vaccine AEs data from EudraVigilance.ResultsOf the 80 059 reports, 1185 (1.5%) were annotated as GBS related. Twenty‐four AEs were identified as having strong association with GBS. The full prediction model, using age, sex, and all 24 AEs achieved an area under the receiver operating characteristic (ROC) curve (AUC) of 85.4% (90% CI: [83.8%, 86.9%]). After excluding the nine (e.g., pruritus, rash, injection site pain) likely underreported AEs, the final AUC became 77.5% (90% CI: [75.5%, 79.6%]). Two hundred and one (0.25%) reports were predicted as of high risk of GBS (predicted probability >25%) and 84 actually developed GBS.ConclusionThe prediction performance demonstrated the potential of developing risk‐prediction models utilizing the VAERS cohort. Excluding the likely underreported AEs sacrificed some prediction power but made the model more interpretable and feasible. The high absolute risk of even a small number of AE combinations suggests the promise of GBS prediction within the VAERS dataset.