Cheung, Joseph

Abstract Introduction While Large Language Models (LLMs) offer generative capabilities, generic models often lack domain-specific grounding, leading to hallucinations in high-stakes contexts. However, LLMs can serve as clinical decision support tools if augmented with specific domain data. This study evaluates the utility of Retrieval-Augmented Generation (RAG) applied to open-source LLMs, utilizing a curated knowledge base of authoritative sleep medicine textbooks, to quantify improvement in performance. Methods We evaluated four open-source models—Llama-70B, Llama-8B, Qwen-14B, and Qwen-235B, linked to a RAG system indexing five standard textbooks, including ICSD-3 TR. Models were tested on sleep board–style MCQs and clinical vignettes diagnostic accuracy. Performance was assessed across nine configurations derived from three factors: RAG pipeline complexity (Plain vs. Complicated), textbook preprocessing (Uncleaned vs. Cleaned/TOC-aligned), and retriever type (Dense-only vs. Hybrid). Metrics included MCQ Accuracy and the rates at which the correct diagnosis appeared as the top result or within the top 5 differential diagnosis lists for vignettes. Results RAG consistently outperformed the no-RAG baseline, yielding absolute performance gains of 5.6%–10.7% for MCQs and 8.1%–10.2% for the top diagnosis in cases across all models. For MCQs, Qwen-235B achieved the highest accuracy of 87.3% (vs. 81.7% baseline) using the Plain RAG configuration with Cleaned (TOC-aligned) Hybrid retrieval. Llama-70B followed with 83.6% in the same configuration. For case vignettes, Qwen-235B achieved a correct top diagnosis rate of 66.3% and a correct diagnosis rate within the top 5 differential diagnosis lists of 90.8% using Cleaned Hybrid retrieval configurations. Hybrid retrieval (Dense + Sparse) consistently surpassed Dense-only methods. Notably, while larger models benefited from Cleaned text, smaller models (e.g., Qwen-14B) achieved higher top diagnosis rates with Uncleaned text (57.1% vs. 51.0%), suggesting a dependency on redundant context in lower-parameter settings. Conclusion RAG significantly enhances knowledge performance of open-source LLMs in sleep medicine. Hybrid retrieval and curated, TOC-aligned knowledge bases yield optimal results for large models, whereas smaller models benefit from the redundancy of uncleaned text. These findings suggest that RAG systems can augment LLMs in domain specific knowledge for clinical decision support. Support (if any)

Abstract Introduction Narcolepsy and idiopathic hypersomnia (IH) are characterized by excessive daytime sleepiness. They can be caused by deficiency of orexin/hypocretin, a neuropeptide that regulates sleep and wakefulness. It is estimated that 1 in 2000 people in the United States of America have been diagnosed with some form of narcolepsy or idiopathic hypersomnia. Around the world, about 3 million people are narcoleptic. Little data are available regarding the lifestyle factors in these patients. In this study, we sought to find whether lifestyle factors, including diet and physical activity are associated with the severity of sleepiness in narcolepsy and idiopathic hypersomnia. Methods We collaborated with “Hypersomnia Foundation” and “Wake Up Narcolepsy” who distributed our survey to their subscribers. The survey consisted of a series of four short subsurveys asking about the substance use, diet (MiniEAT screen), exercise (IPAQ questionnaire), and daytime sleepiness (Epworth Sleepiness Scale). The survey was anonymous and included baseline characteristics, diagnosis (Narcolepsy type 1 (NT1), Narcolepsy type 2 (NT2), or Idiopathic Hypersomnia (IH)). We considered hypersomnolence and moderate to severe hypersomnolence to be present if ESS >10 and ≥15, respectively. Results We received total of 377 individual responses. Of those, 191 responders had IH, 97 had NT1 and 89 had NT2. Median age was 31 (25, 42) and 323 identified as females, 44 males and 10 others. Median ESS was 15 (12, 18), where 285 responders had ESS >10 (76%) and 194 (51%) had ESS >/=15. The ressponders with NT1 had significantly higher median ESS: 17 (12,19), than NT2: 15 (10, 17), or IH: 15(12, 17) responders (p< 0.01). In multiple univariate analyses, none of the baseline characteristics, habits (smoking, alcohol or caffeine), dietary consumption (fruit, vegetable, legumes/nuts/seeds, whole or refined grains, dairy and fish/seafood) or exercise (vigourous, moderate or walking) had significant association with the degree of responders’ subjective sleepiness. Conclusion Most survey responders continue to be excessively sleepy, particularly those with NT1. There were no apparent univariate associations of underlying hypersomnolence with the baseline characteristics, habits, diet and exercise levels. Disorders of central hypersomnolence are of complex nature without easily identifiable modifiers. Support (if any) Mayo Clinic CRISP

Abstract Objective Evaluate how RAG architecture, including corpus structure, retrieval strategy, and pipeline complexity, affects LLM-based medical problem solving and knowledge retrieval in sleep medicine. Materials and Methods We benchmarked four open-source LLMs (Llama-3-8B, Llama -3 -70B, Qwen 2.5-14B, and Qwen 2.5-235B) using a knowledge base of five sleep medicine textbooks. We compared performance across three dimensions: corpus structure (raw text vs table-of-contents aligned.), retrieval strategy (dense embedding vs hybrid sparse-dense), and pipeline complexity (baseline vs augmented). Evaluation metrics included board-style multiple choice question (MCQ) accuracy and clinical case vignette diagnostic ranking. Results RAG improved MCQ accuracy for all models. Llama-8B saw the largest gain of 10.6% (61.8% to 72.4%), while Qwen-235B reached 87.3%. In clinical cases, Llama-8B accuracy dropped by 7.1% when using raw text and dense retrieval due to context noise. This was corrected by using structured hybrid configurations. Hybrid retrieval consistently outperformed dense-only methods. Overall, structured corpora improved primary diagnosis accuracy by 6.1% on average, with Qwen-235B reaching a peak 10.2% increase. Discussion RAG effectiveness depends on the balance between model size and data structure. Large models handle uncurated text well, but smaller models are easily distracted by irrelevant data. Hybrid retrieval is necessary to maintain precision with specialized medical terms. A structured corpus paired with a baseline hybrid pipeline offers the best stability and speed for clinical use. Conclusion Rigorous data curation and hybrid retrieval are as essential as model scale for deploying safe, guideline-compliant AI in sleep medicine.