Cheung, Joseph

Study objectives To evaluate and compare the performance of two proprietary frontier large language models (LLMs), ChatGPT-5 and Grok-4, on diagnostic reasoning and foundational knowledge tasks within the specialty domain of sleep medicine. Methods The models were evaluated on two tasks: case-based reasoning using 79 clinical vignettes from the AASM Case Book of Sleep Medicine and knowledge assessment using 897 multiple-choice questions (MCQs) from board review materials. For vignettes, final diagnosis was scored by concept-level exact match, and differential diagnosis (DDx) was scored on a fixed top-5 output using concept-level matching with synonym normalization to compute precision, recall, and F 1-score. MCQ performance was the proportion correct. Inter-model performance was compared using the Mann–Whitney U test. Results Both models achieved high accuracy for final diagnosis (92.4% for both; 95% CI 86.4, 98.4) and MCQs (ChatGPT-5: 93.0%; Grok-4: 92.8%). However, performance on generating a comprehensive differential diagnosis was suboptimal, with modest F 1-scores for both ChatGPT-5 (0.55 ± 0.20) and Grok-4 (0.59 ± 0.20). There were no statistically significant differences in performance between the two models across any metric ( p > 0.05). Conclusions Frontier LLMs demonstrated high accuracy in sleep medicine tasks requiring knowledge recall and direct pattern recognition but showed more limited performance in complex clinical reasoning tasks such as generating a comprehensive differential diagnosis. These findings suggest that current general-purpose models may be more reliable for focused knowledge support than for broad hypothesis generation. Future studies should evaluate whether domain-adapted models or clinician-in-the-loop workflows can improve real-world diagnostic usefulness and safety.

Abstract Objective Evaluate how RAG architecture, including corpus structure, retrieval strategy, and pipeline complexity, affects LLM-based medical problem solving and knowledge retrieval in sleep medicine. Materials and Methods We benchmarked four open-source LLMs (Llama-3-8B, Llama -3 -70B, Qwen 2.5-14B, and Qwen 2.5-235B) using a knowledge base of five sleep medicine textbooks. We compared performance across three dimensions: corpus structure (raw text vs table-of-contents aligned.), retrieval strategy (dense embedding vs hybrid sparse-dense), and pipeline complexity (baseline vs augmented). Evaluation metrics included board-style multiple choice question (MCQ) accuracy and clinical case vignette diagnostic ranking. Results RAG improved MCQ accuracy for all models. Llama-8B saw the largest gain of 10.6% (61.8% to 72.4%), while Qwen-235B reached 87.3%. In clinical cases, Llama-8B accuracy dropped by 7.1% when using raw text and dense retrieval due to context noise. This was corrected by using structured hybrid configurations. Hybrid retrieval consistently outperformed dense-only methods. Overall, structured corpora improved primary diagnosis accuracy by 6.1% on average, with Qwen-235B reaching a peak 10.2% increase. Discussion RAG effectiveness depends on the balance between model size and data structure. Large models handle uncurated text well, but smaller models are easily distracted by irrelevant data. Hybrid retrieval is necessary to maintain precision with specialized medical terms. A structured corpus paired with a baseline hybrid pipeline offers the best stability and speed for clinical use. Conclusion Rigorous data curation and hybrid retrieval are as essential as model scale for deploying safe, guideline-compliant AI in sleep medicine.

Abstract Introduction Large language models (LLMs), such as ChatGPT-4, have demonstrated promising potential as diagnostic tools across various medical disciplines. In sleep medicine, artificial intelligence tools are already used to analyze physiological sleep data and are being studied for their potential in phenotyping, endotyping, and predicting treatment responses. However, the effectiveness of LLMs in accurately diagnosing sleep disorders based on clinical history has not yet been studied. This study evaluates ChatGPT-4’s diagnostic performance using clinical vignettes. Methods Nineteen clinical cases were selected from the Case Book of Sleep Medicine, Third Edition (AASM, 2019). Each case included patient history, examination findings, and test results. The vignettes were input into ChatGPT-4 using the following standardized prompt: “Based on the provided case details, generate the top 5 differential diagnoses and identify the most likely final diagnosis: (copy and paste clinical vignette).” The model was tasked with generating both differential diagnoses and a final diagnosis for each case. The AI’s results were compared to reference diagnoses from the case book. Differential diagnoses were measured as the number of matches, and accuracy was reported as a percentage. Final diagnoses were scored as 0 (no match), 1 (partial match), or 2 (full match). Results The mean number of AI-generated differential diagnoses matching the AASM case differential diagnoses was 2.79 ± 0.71 (95% CI: 2.45–3.13). The mean accuracy percentage for differential diagnoses was 63.27% ± 15.61% (95% CI: 55.75%–70.79%), with scores ranging from 33.33% to 100%. For final diagnoses, ChatGPT-4 scored a total of 30 out of a possible 38, with a mean score was 1.58 ± 0.61 (95% CI: 1.29–1.87) out of 2, with 74% of cases achieving a full match. Performance was higher in cases with fewer differential diagnoses, whereas accuracy decreased in more complex cases. Conclusion ChatGPT-4 showed moderate to high accuracy in generating both differential and final diagnoses for sleep disorders. These findings suggest that AI could become a valuable clinical decision-support tool in sleep medicine. However, its inconsistent performance in complex cases highlights the need for further refinement and clinical testing. Support (if any)

Abstract Objectives Patients with hypermobile Ehlers–Danlos syndrome (hEDS) and hypermobility spectrum disorders (HSD) experience a wide array of symptoms and system disorders. This study aimed to identify whether differences occurred in 115 self-reported symptoms and comorbidities in patients diagnosed with hEDS or HSD. Methods In this study we analysed self-reported data from an EDS Clinic intake questionnaire in patients diagnosed with hEDS, HSD or no hypermobile conditions. Results From 1 November 2019 to 7 March 2024, the EDS Clinic saw 2088 patients. Using the 2017 diagnostic criteria, 66.5% were diagnosed with HSD (n = 1389), 20.3% with hEDS (n = 423), 10.6% with historic HSD (H-HSD) or localized HSD (L-HSD) (n = 256) (hypermobile controls) and 2.6% were not diagnosed with hEDS, HSD, H-HSD or L-HSD (n = 55) (controls). Symptoms/comorbidities that occurred with high prevalence in both hEDS and HSD included joint pain (hEDS 82.0%, HSD 88.9%), allergy (hEDS 77.0%, HSD 77.0%), subluxations (hEDS 71.2%, HSD 72.6%), brain fog (hEDS 70.0%, HSD 74.7%), headache (hEDS 68.1%, HSD 69.1%), anxiety (hEDS 60.3%, HSD 69.3%), depression (hEDS 52.2%, HSD 58.0%), migraine (hEDS 53.7%, HSD 52.5%), nausea (hEDS 54.6%, HSD 59.5%) and constipation (hEDS 53.0%, HSD 57.2%). In contrast, 9/115 (8%) symptoms/comorbidities were self-reported significantly more often in hEDS but 42/115 (37%) in HSD. hEDS patients reported more symptoms that suggest a defect in collagen, such as dislocation, hernias and rectal prolapse, while HSD patients reported more joint, muscle, allergy, neurological, gastrointestinal, sleep and psychological symptoms/comorbidities. Conclusion Although we found an overlap in some symptoms and comorbidities self-reported by hEDS/HSD patients, such as allergy/atopy, headache/migraine and gastrointestinal symptoms, our findings suggest key differences exist between the two diagnoses, suggesting that hEDS and HSD may be distinct conditions.

Background Postprandial sleepiness refers to the state of excessive drowsiness that occurs after consuming a meal. This transient phenomenon is experienced by many individuals, often leading to decreased productivity and impaired cognitive performance. This study examined the effect of a whole-food plant-based (WFPB) diet on postprandial sleepiness. Methods Ten participants who had postprandial sleepiness who were on a typical American diet transitioned to a WFPB dietary pattern for 21 days. Postprandial sleepiness was assessed using the Epworth Sleepiness Scale (ESS) questionnaire, focusing on Q7 “sitting quietly after lunch without alcohol” as well as Q5 “lying down to rest in the afternoon when circumstances permit.” Results The participants had a mean age of 44 +/− 15 years, with 30% being male. There was a statistically significant reduction in sleepiness measured by ESS Q7 (median change = −1, P = .012) and ESS Q5 (median change = −1.5, P = .008). Moreover, there was a statistically significant decrease in overall ESS scores (median change = −8.0, P = .006), and weight (median change = −2.9 kg, P = .006). Conclusion Participants demonstrated significant improvement in postprandial sleepiness-related situations. Our findings showed that the WFPB diet may have potential benefits for reducing postprandial and overall daytime sleepiness.

Abstract Introduction Regular exercise improves sleep quality by reducing sleep onset latency, extending total sleep duration, and enhancing sleep efficiency, which may alleviate daytime sleepiness. However, the specific impact of exercise on daytime sleepiness remains unclear, with conflicting findings in the literature. We analyzed existing interventional studies which used Epworth Sleepiness Scale (ESS) to measure changes in degree of daytime sleepiness before and after an exercise regimen, allowing us to explore the relationship between exercise and daytime sleepiness. Methods Following PRISMA guidelines, a systematic search was conducted in electronic database of PubMed and Google Scholar from 1991 to the present. Thorough screening and inclusion criteria were applied to identify relevant studies which included ESS assessment before and after an exercise intervention. Also assessed sleep quality by Pittsburgh Sleep Quality Index (PSQI). Results Seven studies met all the inclusion criteria and were included in this study. Exercise exhibited a significantly beneficial impact on sleepiness measured by ESS in four out of seven studies we included in review while two studies showed a marginal reduction in ESS without statistical significance, and one study indicated no change. Additionally, exercise showed an improvement in sleep quality in four out of five studies that included PSQI. Variability in results was noted, influenced by factors such as exercise type, intensity, duration, and timing, as well as individual adherence. Additionally, exercise interventions improved sleep quality, as suggested by reduced scores in PSQI which may indirectly improve daytime sleepiness. Conclusion This analysis of existing interventional studies on exercise and daytime sleepiness suggests exercise can be beneficial for reducing daytime sleepiness and improving sleep quality. While the exact mechanism remains elusive, several factors contribute to the impact of both acute and chronic exercise on daytime sleepiness such as improved sleep quality, regulation of circadian rhythm, neurotransmitters release, stress reduction, increased energy levels, and weight reduction ultimately alleviating daytime sleepiness. Future research will be essential for addressing gaps on how exercise regimen and the underlying mechanisms may prove efficacious on improving daytime sleepiness. Support (if any) Mayo Clinic Florica Research Accelerator for Clinicians Engaged in Research Program

Abstract Introduction Restless legs syndrome (RLS) is a common disorder of unpleasant sensations during inactivity, an urge to move the legs, and hyperarousal. The pathophysiology is incompletely understood. Treatments are not always successful or carry side effects. Methods (Case Report) Results A 45-year-old woman presented with over a decade of RLS and sleep-onset insomnia. The diagnosis was based on the International RLS Study Group criteria. The severity was assessed by the nightly occurrence and Insomnia Severity Index (ISI) of 25. Interfering neurologic or sleep disorders, medications, and abnormal blood work were ruled out. Different dopamine agonists resulted in augmentation. Caffeine was stopped. Polysomnography revealed periodic limb movement with sleep fragmentation. Gabapentin was partially effective and poorly tolerated at high doses, but continued at 600 mg. While using Dextromethorphan (DXM) for an upper respiratory infection, the patient found dramatic improvement of her RLS symptoms. Independently, she discontinued Gabapentin, and continued DXM 30 mg for 6 months, with consistent improvement. ISI decreased to 10. Ultimately, considering its expense, DXM was tapered, and Tramadol initiated. The pathophysiology of RLS is partially understood. Studies showed abnormalities in iron regulation, low adenosine tone, enhanced glutamatergic activity, inflammation, and a genetic predisposition, as contributing factors. Alpha-2-delta calcium channel ligands inhibit the presynaptic glutamate release. Central nervous system iron deficiency may increase glutamatergic tone. Ketamine, an N-methyl-D-aspartate (NMDA) receptor antagonist, inhibits the activation of NMDA receptor by glutamate and decreases glutamate presynaptic release. Ketamine was reported to effectively treat RLS. DXM, another NMDA receptor antagonist, has not been reported to treat RLS in the literature. Additionally, an association between inflammation and RLS was described, and limited data showed anti-inflammatory effect of DXM. Conclusion Patients with RLS suffer from their symptoms’ burden. Approved medications could carry side effects, or augmentation. We report a case of severe RLS with dramatic and sustained improvement with DXM. Support (if any)

Objectives: To assess the prevalence of sleep disturbance among patients evaluated at a clinic for patients afflicted with Post-acute sequelae of COVID-19 (PASC). Methods: Sleep disturbance was assessed with the Patient-Reported Outcomes Measurement Information System-Sleep Disturbance (PROMIS-SD) framework among adult patients of the PASC clinic. Results: Among 312 patients, the mean age was 46.2 years, and 70.2% were women. About 41.0% of patients had no sleep disturbance; sleep disturbance was mild to moderate in 51.3% and severe in 7.7%. PROMIS-SD score was negatively correlated with the time from the initial positive COVID-19 test to the initial consultation in the PASC clinic (Pearson r = −.094; r 2 = .0088). Conclusions: The PROMIS-Sleep Disturbance framework can serve as a tool to assess the burden of sleep disturbances in PASC patients.