Citations

Abstract Current discussion surrounding the clinical capabilities of generative language models (GLMs) predominantly center around multiple-choice question-answer (MCQA) benchmarks derived from clinical licensing examinations. While accepted for human examinees, characteristics unique to GLMs bring into question the validity of such benchmarks. Here, we validate four benchmarks using eight GLMs, ablating for parameter size and reasoning capabilities, validating via prompt permutation three key assumptions that underpin the generalizability of MCQA-based assessments: that knowledge is applied, not memorized, that semantic consistency will lead to consistent answers, and that situations with no answers can be recognized. While large models are more resilient to our perturbations compared to small models, we globally invalidate these assumptions, with implications for reasoning models. Additionally, despite retaining the knowledge, small models are prone to answer memorization. All models exhibit significant failure in null-answer scenarios. We then suggest several adaptations for more robust benchmark designs more reflective of real-world conditions.

Abstract Objective Unlocking clinical information embedded in clinical notes has been hindered to a significant degree by domain-specific and context-sensitive language. Identification of note sections and structural document elements has been shown to improve information extraction and dependent downstream clinical natural language processing (NLP) tasks and applications. This study investigates the viability of a dynamic example selection prompting method to section classification using lightweight, open-source large language models (LLMs) as a practical solution for real-world healthcare clinical NLP systems. Materials and Methods We develop a dynamic few-shot prompting approach to classifying sections where section samples are first embedded using a transformer-based model and deposited in a vector store. During inference, the embedded samples with the most similar contextual embeddings to a given input section text are retrieved from the vector store and inserted into the LLM prompt. We evaluate this technique on two datasets comprising two section schemas, including varying levels of context. We compare the performance to baseline zero-shot and randomly selected few-shot scenarios. Results The dynamic few-shot prompting experiments yielded the highest F1 scores in each of the classification tasks and datasets for all seven of the LLMs included in the evaluation, averaging a macro F1 increase of 39.3% and 21.1% in our primary section classification task over the zero-shot and static few-shot baselines, respectively. Discussion and Conclusion Our results showcase substantial performance improvements imparted by dynamically selecting examples for few-shot LLM prompting, and further improvement by including section context, demonstrating compelling potential for clinical applications.

Abstract Objective SNOMED CT provides a standardized terminology for clinical concepts, allowing cohort queries over heterogeneous clinical data including Electronic Health Records (EHRs). While it is intuitive that missing and inaccurate subtype (or is-a) relations in SNOMED CT reduce the recall and precision of cohort queries, the extent of these impacts has not been formally assessed. This study fills this gap by developing quantitative metrics to measure these impacts and performing statistical analysis on their significance. Material and Methods We used the Optum de-identified COVID-19 Electronic Health Record dataset. We defined micro-averaged and macro-averaged recall and precision metrics to assess the impact of missing and inaccurate is-a relations on cohort queries. Both practical and simulated analyses were performed. Practical analyses involved 407 missing and 48 inaccurate is-a relations confirmed by domain experts, with statistical testing using Wilcoxon signed-rank tests. Simulated analyses used two random sets of 400 is-a relations to simulate missing and inaccurate is-a relations. Results Wilcoxon signed-rank tests from both practical and simulated analyses (P-values < .001) showed that missing is-a relations significantly reduced the micro- and macro-averaged recall, and inaccurate is-a relations significantly reduced the micro- and macro-averaged precision. Discussion The introduced impact metrics can assist SNOMED CT maintainers in prioritizing critical hierarchical defects for quality enhancement. These metrics are generally applicable for assessing the quality impact of a terminology’s subtype hierarchy on its cohort query applications. Conclusion Our results indicate a significant impact of missing and inaccurate is-a relations in SNOMED CT on the recall and precision of cohort queries. Our work highlights the importance of high-quality terminology hierarchy for cohort queries over EHR data and provides valuable insights for prioritizing quality improvements of SNOMED CT's hierarchy.

Abstract Background: Clinical information retrieval (IR) plays a vital role in modern healthcare by facilitating efficient access and analysis of medical literature for clinicians and researchers. This scoping review aims to offer a comprehensive overview of the current state of clinical IR research and identify gaps and potential opportunities for future studies in this field. Objectives: The main objective of this review is to identify and analyze published research on clinical IR, including the methods, techniques, and tools used to retrieve and analyze clinical information from various sources. We aim to provide a comprehensive overview of the current state of clinical IR research and guide future research efforts in this field. Methods: We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines and conducted a thorough search of multiple databases, including Ovid Embase, Ovid Cochrane Central Register of Controlled Trials, Ovid Cochrane Database of Systematic Reviews, Scopus, ACM Digital Library, IEEE Xplore, and Web of Science, from January 1st, 2012, to January 4th, 2023. The screening process involved multiple reviewers, and we included 184 papers for the final review. Results: We conducted a detailed analysis and discussion of various aspects of clinical IR research, including publication year, data sources, methods, techniques, evaluation metrics, shared tasks, and applications. Our analysis revealed key research areas in clinical IR, such as indexing, ranking, and query expansion, and identified opportunities for future research in these areas.

Objective Sudden hearing loss is a frightening symptom that often prompts an urgent or emergent visit to a health care provider. It is frequently, but not universally, accompanied by tinnitus and/or vertigo. Sudden sensorineural hearing loss affects 5 to 27 per 100,000 people annually, with about 66,000 new cases per year in the United States. This guideline update provides evidence‐based recommendations for the diagnosis, management, and follow‐up of patients who present with sudden hearing loss. It focuses on sudden sensorineural hearing loss in adult patients aged 18 and over and primarily on those with idiopathic sudden sensorineural hearing loss. Prompt recognition and management of sudden sensorineural hearing loss may improve hearing recovery and patient quality of life. The guideline update is intended for all clinicians who diagnose or manage adult patients who present with sudden hearing loss. Purpose The purpose of this guideline update is to provide clinicians with evidence‐based recommendations in evaluating patients with sudden hearing loss and sudden sensorineural hearing loss, with particular emphasis on managing idiopathic sudden sensorineural hearing loss. The guideline update group recognized that patients enter the health care system with sudden hearing loss as a nonspecific primary complaint. Therefore, the initial recommendations of this guideline update address distinguishing sensorineural hearing loss from conductive hearing loss at the time of presentation with hearing loss. They also clarify the need to identify rare, nonidiopathic sudden sensorineural hearing loss to help separate those patients from those with idiopathic sudden sensorineural hearing loss, who are the target population for the therapeutic interventions that make up the bulk of the guideline update. By focusing on opportunities for quality improvement, this guideline should improve diagnostic accuracy, facilitate prompt intervention, decrease variations in management, reduce unnecessary tests and imaging procedures, and improve hearing and rehabilitative outcomes for affected patients. Methods Consistent with the American Academy of Otolaryngology–Head and Neck Surgery Foundation’s Clinical Practice Guideline Development Manual, Third Edition , the guideline update group was convened with representation from the disciplines of otolaryngology–head and neck surgery, otology, neurotology, family medicine, audiology, emergency medicine, neurology, radiology, advanced practice nursing, and consumer advocacy. A systematic review of the literature was performed, and the prior clinical practice guideline on sudden hearing loss was reviewed in detail. Key action statements (KASs) were updated with new literature, and evidence profiles were brought up to the current standard. Research needs identified in the original clinical practice guideline and data addressing them were reviewed. Current research needs were identified and delineated. Results The guideline update group made strong recommendations for the following: clinicians should distinguish sensorineural hearing loss from conductive hearing loss when a patient first presents with sudden hearing loss (KAS 1); clinicians should educate patients with sudden sensorineural hearing loss about the natural history of the condition, the benefits and risks of medical interventions, and the limitations of existing evidence regarding efficacy (KAS 7); and clinicians should counsel patients with sudden sensorineural hearing loss who have residual hearing loss and/or tinnitus about the possible benefits of audiological rehabilitation and other supportive measures (KAS 13). These strong recommendations were modified from the initial clinical practice guideline for clarity and timing of intervention. The guideline update group made strong recommendation against the following: clinicians should not order routine computed tomography of the head in the initial evaluation of a patient with presumptive sudden sensorineural hearing loss (KAS 3); clinicians should not obtain routine laboratory tests in patients with sudden sensorineural hearing loss (KAS 5); and clinicians should not routinely prescribe antivirals, thrombolytics, vasodilators, or vasoactive substances to patients with sudden sensorineural hearing loss (KAS 11). The guideline update group made recommendations for the following: clinicians should assess patients with presumptive sudden sensorineural hearing loss through history and physical examination for bilateral sudden hearing loss, recurrent episodes of sudden hearing loss, and/or focal neurologic findings (KAS 2); in patients with sudden hearing loss, clinicians should obtain, or refer to a clinician who can obtain, audiometry as soon as possible (within 14 days of symptom onset) to confirm the diagnosis of sudden sensorineural hearing loss (KAS 4); clinicians should evaluate patients with sudden sensorineural hearing loss for retrocochlear pathology by obtaining a magnetic resonance imaging or auditory brainstem response (KAS 6); clinicians should offer, or refer to a clinician who can offer, intratympanic steroid therapy when patients have incomplete recovery from sudden sensorineural hearing loss 2 to 6 weeks after onset of symptoms (KAS 10); and clinicians should obtain follow‐up audiometric evaluation for patients with sudden sensorineural hearing loss at the conclusion of treatment and within 6 months of completion of treatment (KAS 12). These recommendations were clarified in terms of timing of intervention and audiometry, as well as method of retrocochlear workup. The guideline update group offered the following KASs as options: clinicians may offer corticosteroids as initial therapy to patients with sudden sensorineural hearing loss within 2 weeks of symptom onset (KAS 8); clinicians may offer, or refer to a clinician who can offer, hyperbaric oxygen therapy combined with steroid therapy within 2 weeks of onset of sudden sensorineural hearing loss (KAS 9a); and clinicians may offer, or refer to a clinician who can offer, hyperbaric oxygen therapy combined with steroid therapy as salvage therapy within 1 month of onset of sudden sensorineural hearing loss (KAS 9b). Differences from Prior Guideline Incorporation of new evidence profiles to include quality improvement opportunities, confidence in the evidence, and differences of opinion Included 10 clinical practice guidelines, 29 new systematic reviews, and 36 new randomized controlled trials Highlights the urgency of evaluation and initiation of treatment, if treatment is offered, by emphasizing the time from symptom occurrence Clarification of terminology by changing potentially unclear statements; use of the term sudden sensorineural hearing loss to mean idiopathic sudden sensorineural hearing loss to emphasize that over 90% of sudden sensorineural hearing loss is idiopathic sudden sensorineural hearing loss and to avoid confusion in nomenclature for the reader Changes to the key action statements (KASs) from the original guideline: KAS 1: When a patient first presents with sudden hearing loss, conductive hearing loss should be distinguished from sensorineural. KAS 2: The utility of history and physical examination when assessing for modifying factors is emphasized. KAS 3: The word routine is added to clarify that this statement addresses a nontargeted head computed tomography scan that is often ordered in the emergency room setting for patients presenting with sudden hearing loss. It does not refer to targeted scans such as a temporal bone computed tomography scan to assess for temporal bone pathology. KAS 4: The importance of audiometric confirmation of hearing status as soon as possible and within 14 days of symptom onset is emphasized. KAS 5: New studies were added to confirm the lack of benefit of nontargeted laboratory testing in sudden sensorineural hearing loss. KAS 6: Audiometric follow‐up is excluded as a reasonable workup for retrocochlear pathology. Magnetic resonance imaging, computed tomography scan if magnetic resonance imaging cannot be done, or, secondarily, auditory brainstem response evaluation are the modalities recommended. A time frame for such testing is not specified, nor is it specified which clinician should be ordering this workup; however, it is implied that it would be the general or subspecialty otolaryngologist. KAS 7: The importance of shared decision making is highlighted, and salient points are emphasized. KAS 8: The option for corticosteroid intervention within 2 weeks of symptom onset is emphasized. KAS 9: Changed to KAS 9a and 9b; hyperbaric oxygen therapy remains an option but only when combined with steroid therapy for either initial treatment (9a) or for salvage therapy (9b). The timing is within 2 weeks of onset for initial therapy and within 1 month of onset of sudden sensorineural hearing loss for salvage therapy. KAS 10: Intratympanic steroid therapy for salvage is recommended within 2 to 6 weeks following onset of sudden sensorineural hearing loss. The time to treatment is defined and emphasized. KAS 11: Antioxidants were removed from the list of interventions that the clinical practice guideline recommends against using. KAS 12: Follow‐up audiometry at conclusion of treatment and also within 6 months posttreatment is added. KAS 13: This statement on audiologic rehabilitation includes patients who have residual hearing loss and/or tinnitus who may benefit from treatment. Addition of an algorithm outlining KASs Enhanced emphasis on patient education and shared decision making with tools provided to assist in the same

Objective Sudden hearing loss is a frightening symptom that often prompts an urgent or emergent visit to a health care provider. It is frequently but not universally accompanied by tinnitus and/or vertigo. Sudden sensorineural hearing loss affects 5 to 27 per 100,000 people annually, with about 66,000 new cases per year in the United States. This guideline update provides evidence‐based recommendations for the diagnosis, management, and follow‐up of patients who present with sudden hearing loss. It focuses on sudden sensorineural hearing loss in adult patients aged ≥18 years and primarily on those with idiopathic sudden sensorineural hearing loss. Prompt recognition and management of sudden sensorineural hearing loss may improve hearing recovery and patient quality of life. The guideline update is intended for all clinicians who diagnose or manage adult patients who present with sudden hearing loss. Purpose The purpose of this guideline update is to provide clinicians with evidence‐based recommendations in evaluating patients with sudden hearing loss and sudden sensorineural hearing loss, with particular emphasis on managing idiopathic sudden sensorineural hearing loss. The guideline update group recognized that patients enter the health care system with sudden hearing loss as a nonspecific primary complaint. Therefore, the initial recommendations of this guideline update address distinguishing sensorineural hearing loss from conductive hearing loss at the time of presentation with hearing loss. They also clarify the need to identify rare, nonidiopathic sudden sensorineural hearing loss to help separate those patients from those with idiopathic sudden sensorineural hearing loss, who are the target population for the therapeutic interventions that make up the bulk of the guideline update. By focusing on opportunities for quality improvement, this guideline should improve diagnostic accuracy, facilitate prompt intervention, decrease variations in management, reduce unnecessary tests and imaging procedures, and improve hearing and rehabilitative outcomes for affected patients. Methods Consistent with the American Academy of Otolaryngology–Head and Neck Surgery Foundation's “Clinical Practice Guideline Development Manual, Third Edition” (Rosenfeld et al. Otolaryngol Head Neck Surg . 2013;148[1]:S1‐S55), the guideline update group was convened with representation from the disciplines of otolaryngology–head and neck surgery, otology, neurotology, family medicine, audiology, emergency medicine, neurology, radiology, advanced practice nursing, and consumer advocacy. A systematic review of the literature was performed, and the prior clinical practice guideline on sudden hearing loss was reviewed in detail. Key Action Statements (KASs) were updated with new literature, and evidence profiles were brought up to the current standard. Research needs identified in the original clinical practice guideline and data addressing them were reviewed. Current research needs were identified and delineated. Results The guideline update group made strong recommendations for the following: (KAS 1) Clinicians should distinguish sensorineural hearing loss from conductive hearing loss when a patient first presents with sudden hearing loss. (KAS 7) Clinicians should educate patients with sudden sensorineural hearing loss about the natural history of the condition, the benefits and risks of medical interventions, and the limitations of existing evidence regarding efficacy. (KAS 13) Clinicians should counsel patients with sudden sensorineural hearing loss who have residual hearing loss and/or tinnitus about the possible benefits of audiologic rehabilitation and other supportive measures. These strong recommendations were modified from the initial clinical practice guideline for clarity and timing of intervention. The guideline update group made strong recommendations against the following: (KAS 3) Clinicians should not order routine computed tomography of the head in the initial evaluation of a patient with presumptive sudden sensorineural hearing loss. (KAS 5) Clinicians should not obtain routine laboratory tests in patients with sudden sensorineural hearing loss. (KAS 11) Clinicians should not routinely prescribe antivirals, thrombolytics, vasodilators, or vasoactive substances to patients with sudden sensorineural hearing loss. The guideline update group made recommendations for the following: (KAS 2) Clinicians should assess patients with presumptive sudden sensorineural hearing loss through history and physical examination for bilateral sudden hearing loss, recurrent episodes of sudden hearing loss, and/or focal neurologic findings. (KAS 4) In patients with sudden hearing loss, clinicians should obtain, or refer to a clinician who can obtain, audiometry as soon as possible (within 14 days of symptom onset) to confirm the diagnosis of sudden sensorineural hearing loss. (KAS 6) Clinicians should evaluate patients with sudden sensorineural hearing loss for retrocochlear pathology by obtaining magnetic resonance imaging or auditory brainstem response. (KAS 10) Clinicians should offer, or refer to a clinician who can offer, intratympanic steroid therapy when patients have incomplete recovery from sudden sensorineural hearing loss 2 to 6 weeks after onset of symptoms. (KAS 12) Clinicians should obtain follow‐up audiometric evaluation for patients with sudden sensorineural hearing loss at the conclusion of treatment and within 6 months of completion of treatment. These recommendations were clarified in terms of timing of intervention and audiometry and method of retrocochlear workup. The guideline update group offered the following KASs as options : (KAS 8) Clinicians may offer corticosteroids as initial therapy to patients with sudden sensorineural hearing loss within 2 weeks of symptom onset. (KAS 9a) Clinicians may offer, or refer to a clinician who can offer, hyperbaric oxygen therapy combined with steroid therapy within 2 weeks of onset of sudden sensorineural hearing loss. (KAS 9b) Clinicians may offer, or refer to a clinician who can offer, hyperbaric oxygen therapy combined with steroid therapy as salvage therapy within 1 month of onset of sudden sensorineural hearing loss. Differences from Prior Guideline Incorporation of new evidence profiles to include quality improvement opportunities, confidence in the evidence, and differences of opinion Included 10 clinical practice guidelines, 29 new systematic reviews, and 36 new randomized controlled trials Highlights the urgency of evaluation and initiation of treatment, if treatment is offered, by emphasizing the time from symptom occurrence Clarification of terminology by changing potentially unclear statements; use of the term sudden sensorineural hearing loss to mean idiopathic sudden sensorineural hearing loss to emphasize that >90% of sudden sensorineural hearing loss is idiopathic sudden sensorineural hearing loss and to avoid confusion in nomenclature for the reader Changes to the KASs from the original guideline: KAS 1—When a patient first presents with sudden hearing loss, conductive hearing loss should be distinguished from sensorineural. KAS 2—The utility of history and physical examination when assessing for modifying factors is emphasized. KAS 3—The word “routine” is added to clarify that this statement addresses nontargeted head computerized tomography scan that is often ordered in the emergency room setting for patients presenting with sudden hearing loss. It does not refer to targeted scans, such as temporal bone computerized tomography scan, to assess for temporal bone pathology. KAS 4—The importance of audiometric confirmation of hearing status as soon as possible and within 14 days of symptom onset is emphasized. KAS 5—New studies were added to confirm the lack of benefit of nontargeted laboratory testing in sudden sensorineural hearing loss. KAS 6—Audiometric follow‐up is excluded as a reasonable workup for retrocochlear pathology. Magnetic resonance imaging, computerized tomography scan if magnetic resonance imaging cannot be done, and, secondarily, auditory brainstem response evaluation are the modalities recommended. A time frame for such testing is not specified, nor is it specified which clinician should be ordering this workup; however, it is implied that it would be the general or subspecialty otolaryngologist. KAS 7—The importance of shared decision making is highlighted, and salient points are emphasized. KAS 8—The option for corticosteroid intervention within 2 weeks of symptom onset is emphasized. KAS 9—Changed to KAS 9A and 9B. Hyperbaric oxygen therapy remains an option but only when combined with steroid therapy for either initial treatment (9A) or salvage therapy (9B). The timing of initial therapy is within 2 weeks of onset, and that of salvage therapy is within 1 month of onset of sudden sensorineural hearing loss. KAS 10—Intratympanic steroid therapy for salvage is recommended within 2 to 6 weeks following onset of sudden sensorineural hearing loss. The time to treatment is defined and emphasized. KAS 11—Antioxidants were removed from the list of interventions that the clinical practice guideline recommends against using. KAS 12—Follow‐up audiometry at conclusion of treatment and also within 6 months posttreatment is added. KAS 13—This statement on audiologic rehabilitation includes patients who have residual hearing loss and/or tinnitus who may benefit from treatment. Addition of an algorithm outlining KASs Enhanced emphasis on patient education and shared decision making with tools provided to assist in same