Miao, Jing

Background Professional opinion polling has become a popular means of seeking advice for complex nephrology questions in the #AskRenal community on X. ChatGPT is a large language model with remarkable problem-solving capabilities, but its ability to provide solutions for real-world clinical scenarios remains unproven. This study seeks to evaluate how closely ChatGPT's responses align with current prevailing medical opinions in nephrology. Methods Nephrology polls from X were submitted to ChatGPT-4, which generated answers without prior knowledge of the poll outcomes. Its responses were compared to the poll results (inter-rater) and a second set of responses given after a one-week interval (intra-rater) using Cohen's kappa statistic (κ). Subgroup analysis was performed based on question subject matter. Results Our analysis comprised two rounds of testing ChatGPT on 271 nephrology-related questions. In the first round, ChatGPT's responses agreed with poll results for 163 of the 271 questions (60.2%; κ = 0.42, 95% CI: 0.38–0.46). In the second round, conducted to assess reproducibility, agreement improved slightly to 171 out of 271 questions (63.1%; κ = 0.46, 95% CI: 0.42–0.50). Comparison of ChatGPT's responses between the two rounds demonstrated high internal consistency, with agreement in 245 out of 271 responses (90.4%; κ = 0.86, 95% CI: 0.82–0.90). Subgroup analysis revealed stronger performance in the combined areas of homeostasis, nephrolithiasis, and pharmacology (κ = 0.53, 95% CI: 0.47–0.59 in both rounds), compared to other nephrology subfields. Conclusion ChatGPT-4 demonstrates modest capability in replicating prevailing professional opinion in nephrology polls overall, with varying performance levels between question topics and excellent internal consistency. This study provides insights into the potential and limitations of using ChatGPT in medical decision making.

Chronic kidney disease (CKD) patients can benefit from personalized education on lifestyle and nutrition management strategies to enhance healthcare outcomes. The potential use of chatbots, introduced in 2022, as a tool for educating CKD patients has been explored. A set of 15 questions on lifestyle modification and nutrition, derived from a thorough review of three specific KDIGO guidelines, were developed and posed in various formats, including original, paraphrased with different adverbs, incomplete sentences, and misspellings. Four versions of AI were used to answer these questions: ChatGPT 3.5 (March and September 2023 versions), ChatGPT 4, and Bard AI. Additionally, 20 questions on lifestyle modification and nutrition were derived from the NKF KDOQI guidelines for nutrition in CKD (2020 Update) and answered by four versions of chatbots. Nephrologists reviewed all answers for accuracy. ChatGPT 3.5 produced largely accurate responses across the different question complexities, with occasional misleading statements from the March version. The September 2023 version frequently cited its last update as September 2021 and did not provide specific references, while the November 2023 version did not provide any misleading information. ChatGPT 4 presented answers similar to 3.5 but with improved reference citations, though not always directly relevant. Bard AI, while largely accurate with pictorial representation at times, occasionally produced misleading statements and had inconsistent reference quality, although an improvement was noted over time. Bing AI from November 2023 had short answers without detailed elaboration and sometimes just answered “YES”. Chatbots demonstrate potential as personalized educational tools for CKD that utilize layman’s terms, deliver timely and rapid responses in multiple languages, and offer a conversational pattern advantageous for patient engagement. Despite improvements observed from March to November 2023, some answers remained potentially misleading. ChatGPT 4 offers some advantages over 3.5, although the differences are limited. Collaboration between healthcare professionals and AI developers is essential to improve healthcare delivery and ensure the safe incorporation of chatbots into patient care.

Background: Despite the prevalence and incidence of kidney stones progressively increasing worldwide, public awareness of this condition remains unclear. Understanding trends of awareness can assist healthcare professionals and policymakers in planning and implementing targeted health interventions. This study investigated online search interest in “kidney stone” by analyzing Google Trends, focusing on stationarity of the trends and predicting future trends. Methods: We performed time series analysis on worldwide Google monthly search data from January 2004 to November 2023. The Augmented Dickey–Fuller (ADF) test was used to assess the stationarity of the data, with a p-value below 0.05 indicating stationarity. Time series forecasting was performed using the autoregressive integrated moving average to predict future trends. Results: The highest search interest for “kidney stone” (score 100) was in August 2022, while the lowest was in December 2007 (score 36). As of November 2023, search interest remained high, at 92. The ADF test was significant (p = 0.023), confirming data stationarity. The time series forecasting projected continued high public interest, likely reflecting ongoing concern and awareness. Notably, diverse regions such as Iran, the Philippines, Ecuador, the United States, and Nepal showed significant interest, suggesting widespread awareness of nephrolithiasis. Conclusion: This study highlighted that “kidney stone” is a consistently relevant health issue globally. The increase and stationarity of search trends, the forecasted sustained interest, and diverse regional interest emphasize the need for collaborative research and educational initiatives. This study’s analysis serves as a valuable tool for shaping future healthcare policies and research directions in addressing nephrolithiasis related health challenges.