Mulhall, J

Abstract Introduction An iteration of artificial intelligence (AI), large language models (LLMs), such as ChatGPT, have recently gained popularity as a potential educational resource for patients, students, and researchers. LLMs utilize deep learning algorithms to predict the next word in a sequence based on a data library of 300 billion words. Recent studies have shown that while LLMs provide useful information, they are limited due to producing false and nonsensical information. A focus of studies has been on the utility of ChatGPT as an educational resource to patients and learners; however, no study has investigated the utilization of ChatGPT to write a scientific review. AI- generated systematic reviews would be valuable to the dynamic field of urology, especially in terms of research and innovation. Objective Considering the limitations of ChatGPT in performing data analysis, the aim of this project was to assess the utility of AI in constructing introductions of a systematic review on urologic issues and to compare these AI-generated introductions to published introductions written by humans. Methods ChatGPT (4.0), available at www.chat.openai.com, was used to construct three introductions to three different systematic reviews on the following topics: testosterone therapy and cardiovascular events, safety and efficacy of noninvasive therapies for Peyronie’s disease, and erectile dysfunction after radical prostatectomy. Introductions from published systematic reviews in each of the above topics were also obtained. Five different experts in these topics, who were blinded to which introductions were AI generated, reviewed then assessed the introductions for the following parameters: completeness, engagement, accuracy, logic, and clarity of objective. Introductions were assessed for baseline literary differences as well. Results There were no significant differences in the words, characters, number of sentences, and length of sentences between AI-generated introductions compared to those written by humans. There was a statistically significant difference in word length, with AI having a lower average of word length. AI scored significantly higher on average in the categories of completeness and engagement. We did not observe a significant difference between the two groups in the other categories. These observed findings demonstrate that AI can synthesize urologic literature (sexual medicine/men’s health) and generate introductions of a systematic review that are minimally noninferior and possibly superior regarding engagement and completeness. The limitations from this study include a small sample size (5) and the exclusive focus on AI-generated introductions rather than other AI-generated sections of a systematic review. Conclusions Our study reveals that ChatGPT, as an LLM, can provide a noninferior, possibly superior, introduction to a systematic review on topics of urologic sexual medicine and men’s health. As more facets of AI are developed, such as data integration and analysis, AI could be used to generate entire systematic reviews noninferior to those written by humans. Ethical principles of using AI in scientific literature should be examined and discussed prior to the development of more enhanced LLMs. Disclosure No.

ABSTRACT Introduction Despite an increasing number of publications on Peyronie's disease (PD), evidence-based clinical decision making remains challenging due to the small number of well-designed clinical trials. The quality of available PD literature has not been assessed to date. Objective We sought to evaluate the quality of the methodology and reporting of systematic reviews and meta-analyses in PD literature Methods Study protocol registration was performed on the Open Science Framework platform. In January 2021, a systematic electronic search of the Medline/PubMed, Embase, Ovid, Scopus, Joanne Briggs Institute and Cochrane databases was performed. Search terms included “Peyronie's disease” and “systematic review OR meta-analysis OR meta analysis”. Eligibility criteria were relevance to Peyronie's disease and specification of “systematic review” or “meta-analysis” in the title or abstract. A database was created to summarize the relevant methodological and reporting criteria for each article. The outcomes included review type, authorship, journal, publication date, “A MeaSurement Tool to Assess systematic Reviews” (AMSTAR-2) score and “Preferred Reporting Items for Systematic Reviews and Meta-Analyses” (PRISMA) score. Risk of bias was assessed using the ROBIS tool. Data were tabulated and reported as means with standard deviation, median with interquartile range (IQR) and t-testing as appropriate. Strength of association between variables was calculated using the Pearson correlation coefficient. Statistical analyses were performed on RStudio (version 1.4.1106). Results From 1974 to 2021, 340 articles were identified. After review 23 full length articles were included. Seventeen were systematic reviews, 4 meta-analyses and 2 were combined. Twenty-one (87.5%) articles were published after the year 2009 when the PRISMA guidelines were published. There was median 53.8% (IQR 38.2%) adherence to AMSTAR-2 criteria and 66.7% (IQR 38.9%) adherence to PRISMA criteria. The overall AMSTAR confidence rating was Critically Low in 15/23 studies, indicating more than one critical flaw. Correlation analysis revealed very high positive association between AMSTAR 2 and PRISMA adherence (+0.97). ROBIS revealed “High” concern regarding methods used to collect data and appraise 17/23 studies (73.9%), and “High” concern regarding synthesis and findings in 14/23 studies (60.9%). Conclusions Transparent reporting of methodology and findings in SRs and MAs is crucial to maintaining research integrity and allowing for accurate interpretation of results. Despite an increasing number of publications in PD, many of these studies fail to meet accepted methodological criteria for reporting. More high quality randomized controlled trials in PD are necessary; while systematic reviews and meta-analyses should focus on these high-quality studies with uniform use of accepted reporting guidelines. Disclosure No