Citations

Abstract Introduction While Large Language Models (LLMs) offer generative capabilities, generic models often lack domain-specific grounding, leading to hallucinations in high-stakes contexts. However, LLMs can serve as clinical decision support tools if augmented with specific domain data. This study evaluates the utility of Retrieval-Augmented Generation (RAG) applied to open-source LLMs, utilizing a curated knowledge base of authoritative sleep medicine textbooks, to quantify improvement in performance. Methods We evaluated four open-source models—Llama-70B, Llama-8B, Qwen-14B, and Qwen-235B, linked to a RAG system indexing five standard textbooks, including ICSD-3 TR. Models were tested on sleep board–style MCQs and clinical vignettes diagnostic accuracy. Performance was assessed across nine configurations derived from three factors: RAG pipeline complexity (Plain vs. Complicated), textbook preprocessing (Uncleaned vs. Cleaned/TOC-aligned), and retriever type (Dense-only vs. Hybrid). Metrics included MCQ Accuracy and the rates at which the correct diagnosis appeared as the top result or within the top 5 differential diagnosis lists for vignettes. Results RAG consistently outperformed the no-RAG baseline, yielding absolute performance gains of 5.6%–10.7% for MCQs and 8.1%–10.2% for the top diagnosis in cases across all models. For MCQs, Qwen-235B achieved the highest accuracy of 87.3% (vs. 81.7% baseline) using the Plain RAG configuration with Cleaned (TOC-aligned) Hybrid retrieval. Llama-70B followed with 83.6% in the same configuration. For case vignettes, Qwen-235B achieved a correct top diagnosis rate of 66.3% and a correct diagnosis rate within the top 5 differential diagnosis lists of 90.8% using Cleaned Hybrid retrieval configurations. Hybrid retrieval (Dense + Sparse) consistently surpassed Dense-only methods. Notably, while larger models benefited from Cleaned text, smaller models (e.g., Qwen-14B) achieved higher top diagnosis rates with Uncleaned text (57.1% vs. 51.0%), suggesting a dependency on redundant context in lower-parameter settings. Conclusion RAG significantly enhances knowledge performance of open-source LLMs in sleep medicine. Hybrid retrieval and curated, TOC-aligned knowledge bases yield optimal results for large models, whereas smaller models benefit from the redundancy of uncleaned text. These findings suggest that RAG systems can augment LLMs in domain specific knowledge for clinical decision support. Support (if any)

Abstract Introduction Narcolepsy and idiopathic hypersomnia (IH) are characterized by excessive daytime sleepiness. They can be caused by deficiency of orexin/hypocretin, a neuropeptide that regulates sleep and wakefulness. It is estimated that 1 in 2000 people in the United States of America have been diagnosed with some form of narcolepsy or idiopathic hypersomnia. Around the world, about 3 million people are narcoleptic. Little data are available regarding the lifestyle factors in these patients. In this study, we sought to find whether lifestyle factors, including diet and physical activity are associated with the severity of sleepiness in narcolepsy and idiopathic hypersomnia. Methods We collaborated with “Hypersomnia Foundation” and “Wake Up Narcolepsy” who distributed our survey to their subscribers. The survey consisted of a series of four short subsurveys asking about the substance use, diet (MiniEAT screen), exercise (IPAQ questionnaire), and daytime sleepiness (Epworth Sleepiness Scale). The survey was anonymous and included baseline characteristics, diagnosis (Narcolepsy type 1 (NT1), Narcolepsy type 2 (NT2), or Idiopathic Hypersomnia (IH)). We considered hypersomnolence and moderate to severe hypersomnolence to be present if ESS >10 and ≥15, respectively. Results We received total of 377 individual responses. Of those, 191 responders had IH, 97 had NT1 and 89 had NT2. Median age was 31 (25, 42) and 323 identified as females, 44 males and 10 others. Median ESS was 15 (12, 18), where 285 responders had ESS >10 (76%) and 194 (51%) had ESS >/=15. The ressponders with NT1 had significantly higher median ESS: 17 (12,19), than NT2: 15 (10, 17), or IH: 15(12, 17) responders (p< 0.01). In multiple univariate analyses, none of the baseline characteristics, habits (smoking, alcohol or caffeine), dietary consumption (fruit, vegetable, legumes/nuts/seeds, whole or refined grains, dairy and fish/seafood) or exercise (vigourous, moderate or walking) had significant association with the degree of responders’ subjective sleepiness. Conclusion Most survey responders continue to be excessively sleepy, particularly those with NT1. There were no apparent univariate associations of underlying hypersomnolence with the baseline characteristics, habits, diet and exercise levels. Disorders of central hypersomnolence are of complex nature without easily identifiable modifiers. Support (if any) Mayo Clinic CRISP

Study objectives To evaluate and compare the performance of two proprietary frontier large language models (LLMs), ChatGPT-5 and Grok-4, on diagnostic reasoning and foundational knowledge tasks within the specialty domain of sleep medicine. Methods The models were evaluated on two tasks: case-based reasoning using 79 clinical vignettes from the AASM Case Book of Sleep Medicine and knowledge assessment using 897 multiple-choice questions (MCQs) from board review materials. For vignettes, final diagnosis was scored by concept-level exact match, and differential diagnosis (DDx) was scored on a fixed top-5 output using concept-level matching with synonym normalization to compute precision, recall, and F 1-score. MCQ performance was the proportion correct. Inter-model performance was compared using the Mann–Whitney U test. Results Both models achieved high accuracy for final diagnosis (92.4% for both; 95% CI 86.4, 98.4) and MCQs (ChatGPT-5: 93.0%; Grok-4: 92.8%). However, performance on generating a comprehensive differential diagnosis was suboptimal, with modest F 1-scores for both ChatGPT-5 (0.55 ± 0.20) and Grok-4 (0.59 ± 0.20). There were no statistically significant differences in performance between the two models across any metric ( p > 0.05). Conclusions Frontier LLMs demonstrated high accuracy in sleep medicine tasks requiring knowledge recall and direct pattern recognition but showed more limited performance in complex clinical reasoning tasks such as generating a comprehensive differential diagnosis. These findings suggest that current general-purpose models may be more reliable for focused knowledge support than for broad hypothesis generation. Future studies should evaluate whether domain-adapted models or clinician-in-the-loop workflows can improve real-world diagnostic usefulness and safety.

Abstract Objective Evaluate how RAG architecture, including corpus structure, retrieval strategy, and pipeline complexity, affects LLM-based medical problem solving and knowledge retrieval in sleep medicine. Materials and Methods We benchmarked four open-source LLMs (Llama-3-8B, Llama -3 -70B, Qwen 2.5-14B, and Qwen 2.5-235B) using a knowledge base of five sleep medicine textbooks. We compared performance across three dimensions: corpus structure (raw text vs table-of-contents aligned.), retrieval strategy (dense embedding vs hybrid sparse-dense), and pipeline complexity (baseline vs augmented). Evaluation metrics included board-style multiple choice question (MCQ) accuracy and clinical case vignette diagnostic ranking. Results RAG improved MCQ accuracy for all models. Llama-8B saw the largest gain of 10.6% (61.8% to 72.4%), while Qwen-235B reached 87.3%. In clinical cases, Llama-8B accuracy dropped by 7.1% when using raw text and dense retrieval due to context noise. This was corrected by using structured hybrid configurations. Hybrid retrieval consistently outperformed dense-only methods. Overall, structured corpora improved primary diagnosis accuracy by 6.1% on average, with Qwen-235B reaching a peak 10.2% increase. Discussion RAG effectiveness depends on the balance between model size and data structure. Large models handle uncurated text well, but smaller models are easily distracted by irrelevant data. Hybrid retrieval is necessary to maintain precision with specialized medical terms. A structured corpus paired with a baseline hybrid pipeline offers the best stability and speed for clinical use. Conclusion Rigorous data curation and hybrid retrieval are as essential as model scale for deploying safe, guideline-compliant AI in sleep medicine.

Abstract Introduction Large language models (LLMs), such as ChatGPT-4, have demonstrated promising potential as diagnostic tools across various medical disciplines. In sleep medicine, artificial intelligence tools are already used to analyze physiological sleep data and are being studied for their potential in phenotyping, endotyping, and predicting treatment responses. However, the effectiveness of LLMs in accurately diagnosing sleep disorders based on clinical history has not yet been studied. This study evaluates ChatGPT-4’s diagnostic performance using clinical vignettes. Methods Nineteen clinical cases were selected from the Case Book of Sleep Medicine, Third Edition (AASM, 2019). Each case included patient history, examination findings, and test results. The vignettes were input into ChatGPT-4 using the following standardized prompt: “Based on the provided case details, generate the top 5 differential diagnoses and identify the most likely final diagnosis: (copy and paste clinical vignette).” The model was tasked with generating both differential diagnoses and a final diagnosis for each case. The AI’s results were compared to reference diagnoses from the case book. Differential diagnoses were measured as the number of matches, and accuracy was reported as a percentage. Final diagnoses were scored as 0 (no match), 1 (partial match), or 2 (full match). Results The mean number of AI-generated differential diagnoses matching the AASM case differential diagnoses was 2.79 ± 0.71 (95% CI: 2.45–3.13). The mean accuracy percentage for differential diagnoses was 63.27% ± 15.61% (95% CI: 55.75%–70.79%), with scores ranging from 33.33% to 100%. For final diagnoses, ChatGPT-4 scored a total of 30 out of a possible 38, with a mean score was 1.58 ± 0.61 (95% CI: 1.29–1.87) out of 2, with 74% of cases achieving a full match. Performance was higher in cases with fewer differential diagnoses, whereas accuracy decreased in more complex cases. Conclusion ChatGPT-4 showed moderate to high accuracy in generating both differential and final diagnoses for sleep disorders. These findings suggest that AI could become a valuable clinical decision-support tool in sleep medicine. However, its inconsistent performance in complex cases highlights the need for further refinement and clinical testing. Support (if any)

Abstract Objectives Patients with hypermobile Ehlers–Danlos syndrome (hEDS) and hypermobility spectrum disorders (HSD) experience a wide array of symptoms and system disorders. This study aimed to identify whether differences occurred in 115 self-reported symptoms and comorbidities in patients diagnosed with hEDS or HSD. Methods In this study we analysed self-reported data from an EDS Clinic intake questionnaire in patients diagnosed with hEDS, HSD or no hypermobile conditions. Results From 1 November 2019 to 7 March 2024, the EDS Clinic saw 2088 patients. Using the 2017 diagnostic criteria, 66.5% were diagnosed with HSD (n = 1389), 20.3% with hEDS (n = 423), 10.6% with historic HSD (H-HSD) or localized HSD (L-HSD) (n = 256) (hypermobile controls) and 2.6% were not diagnosed with hEDS, HSD, H-HSD or L-HSD (n = 55) (controls). Symptoms/comorbidities that occurred with high prevalence in both hEDS and HSD included joint pain (hEDS 82.0%, HSD 88.9%), allergy (hEDS 77.0%, HSD 77.0%), subluxations (hEDS 71.2%, HSD 72.6%), brain fog (hEDS 70.0%, HSD 74.7%), headache (hEDS 68.1%, HSD 69.1%), anxiety (hEDS 60.3%, HSD 69.3%), depression (hEDS 52.2%, HSD 58.0%), migraine (hEDS 53.7%, HSD 52.5%), nausea (hEDS 54.6%, HSD 59.5%) and constipation (hEDS 53.0%, HSD 57.2%). In contrast, 9/115 (8%) symptoms/comorbidities were self-reported significantly more often in hEDS but 42/115 (37%) in HSD. hEDS patients reported more symptoms that suggest a defect in collagen, such as dislocation, hernias and rectal prolapse, while HSD patients reported more joint, muscle, allergy, neurological, gastrointestinal, sleep and psychological symptoms/comorbidities. Conclusion Although we found an overlap in some symptoms and comorbidities self-reported by hEDS/HSD patients, such as allergy/atopy, headache/migraine and gastrointestinal symptoms, our findings suggest key differences exist between the two diagnoses, suggesting that hEDS and HSD may be distinct conditions.

Background Postprandial sleepiness refers to the state of excessive drowsiness that occurs after consuming a meal. This transient phenomenon is experienced by many individuals, often leading to decreased productivity and impaired cognitive performance. This study examined the effect of a whole-food plant-based (WFPB) diet on postprandial sleepiness. Methods Ten participants who had postprandial sleepiness who were on a typical American diet transitioned to a WFPB dietary pattern for 21 days. Postprandial sleepiness was assessed using the Epworth Sleepiness Scale (ESS) questionnaire, focusing on Q7 “sitting quietly after lunch without alcohol” as well as Q5 “lying down to rest in the afternoon when circumstances permit.” Results The participants had a mean age of 44 +/− 15 years, with 30% being male. There was a statistically significant reduction in sleepiness measured by ESS Q7 (median change = −1, P = .012) and ESS Q5 (median change = −1.5, P = .008). Moreover, there was a statistically significant decrease in overall ESS scores (median change = −8.0, P = .006), and weight (median change = −2.9 kg, P = .006). Conclusion Participants demonstrated significant improvement in postprandial sleepiness-related situations. Our findings showed that the WFPB diet may have potential benefits for reducing postprandial and overall daytime sleepiness.