Accuracy, readability, and health information quality of AI chatbot responses for patient education on intermittent fasting in chronic kidney disease: a prospective structured evaluation
Frontiers in Nephrology | May 29, 2026 | Volume: 6 | Issue: No issue available | Pages: No page available
Background Intermittent fasting (IF) represents a dietary strategy with known metabolic benefits, however its safety and applicability in patients with chronic kidney disease (CKD) remains poorly understood. AI chatbots’ reliability for clinical and dietary guidance in clinically complex areas remains underexplored. No study has examined AI performance on IF in patients with CKD. This study evaluated the accuracy, consistency, readability, and quality of responses generated by ChatGPT-5 and Claude 4.5 Sonnet regarding IF in CKD. Methods Twenty-five patient-centered questions spanning seven clinical domains were developed by two board-certified nephrologists and submitted independently to ChatGPT-5 and Claude 4.5 Sonnet twice per model. Prompts were submitted under standard and unmodified user conditions without customized system instructions to represent a zero-shot evaluation that is illustrative of typical patient use. Responses were evaluated for accuracy, consistency (comparatively between responses), and appropriateness by the evaluators. Response readability and quality were assessed using the Flesch-Kincaid Grade Level (FKGL) formula and the DISCERN instrument respectively. Results Both ChatGPT-5 and Claude 4.5 Sonnet demonstrated identical accuracy scores during the first question prompts with notable improvement upon re-prompting after 24-hours. Both models demonstrated an identical internal consistency at 90%. ChatGPT-5 produced responses at the middle school level, better aligned with the US adult health literacy benchmark, while Claude 4.5 Sonnet generated responses at the high school level. DISCERN scoring was comparable between chatbots indicative of good response quality. Both chatbots had similar performance regarding individual subdomains. Conclusions Both chatbots illustrated clinically accurate and cautious responses to questions regarding IF in CKD. ChatGPT-5 responses were better suited to the average health literacy of the general United States adult population. Despite our findings, we do not recommend either chatbot be used as the primary source for clinical counseling regarding IF in CKD. It is important for clinicians to help to guide patients toward evidence-based clinical information.