Bedrick, Steven

The 2025 Voice AI Symposium represented a transition from conceptual research to clinical implementation in vocal biomarker science. Hosted by the NIH-funded Bridge2AI-Voice consortium, the meeting convened global experts to address the methodological, ethical, and translational challenges of integrating voice-based artificial intelligence (AI) into healthcare. This mini-review synthesizes symposium insights across six domains: multimodal integration, FAIR (Findable, Accessible, Interoperable, Reusable) and CARE (Collective Benefit, Authority to Control, Responsibility, Ethics) data governance, clinical translation, interdisciplinary training, and cross-sector innovation. Research presented demonstrated voice as a latent, multimodal biomarker reflecting neurological, cardiopulmonary, and psychological states, while discussions emphasized ethical data practices and human-centered design. The implementation-focused panels underscored the importance of workflow alignment and usability for adoption in real-world care. Collectively, the symposium reflects a field advancing toward translational readiness and ethical accountability, positioning voice AI as a scalable, inclusive tool for next-generation healthcare.

Abstract Importance Voice-based health technologies are growing rapidly, but they lack standardized terminology, which hinders interdisciplinary collaboration, research quality, and clinical translation. Objective The objective of this work is to develop universally accepted definitions in the rapidly evolving field of vocal biomarkers, as part of the VOCAL ( V ocal Biomarker Guidelines for O ntology, C lassification, A pplication, and L ogistics) initiative, a structured, international consensus-based framework that aims to provide standards, and guidelines. Design VOCAL is a rigorous, international, multi-stage consensus-building study conducted in 2024-2025. Setting Multi-institutional collaboration between representatives from the Bridge2AI-Voice Consortium (North America) and the eVoiceNet Network (European Union), culminating in an in-person workshop at the 2025 Bridge2AI Voice Symposium. Participants A group of 24 international experts in medicine, clinical research, speech and language, audio signal processing, statistics, methodology, regulation, ethics. Methods VOCAL’s iterative process involved five rounds of review, feedback, and an in-person workshop at the international 2025 Bridge2AI Voice Symposium, ensuring the incorporation of diverse perspectives and achieving a robust agreement on the proposed definitions. Main outcomes and measures Consensus-based definitions for vocal biomarkers, spanning from broad concepts (biomarker, digital biomarker, vocal biomarker) to domain-specific measures (cardio-respiratory acoustic, voice, speech/articulatory, cognitive/language). Results A hierarchical continuum model of vocal biomarkers was established. We first distinguished between the concepts of vocal measures and vocal biomarkers. We then defined terms from broad, overarching concepts (Level 0: Biomarker, Digital Biomarker, Vocal Biomarker) to more specific physiological and cognitive domains (Level 1: Cardio-Respiratory Acoustic; Level 2: Voice; Level 3: Speech/Articulatory; Level 4: Cognitive/Language, including linguistic and paralinguistic subtypes). Conclusions and Relevance This work provides a shared vocabulary that is essential for fostering communication through interdisciplinary collaboration, improving the quality and efficiency of research and development, and ensuring the ethical, reliable, and scalable deployment of future voice-based health technologies. It lays foundational groundwork for upcoming guidelines and standards, which are crucial for advancing the field of vocal biomarkers into widespread clinical utility. Key points Question Can we establish an international consensus on the definitions related to vocal biomarkers? Findings Through a multi-stage international consensus process involving expert representatives from the Bridge2AI-Voice Consortium and eVoiceNet Network, a hierarchical model of vocal biomarkers was developed, defining terms from broad concepts (biomarker, digital biomarker, vocal biomarker) to specific physiological and cognitive domains (cardio-respiratory acoustic, voice, speech/articulatory, cognitive/language). Meaning Standardized definitions for vocal biomarkers provide an essential shared vocabulary for interdisciplinary collaboration and lay the foundational groundwork for future guidelines needed to advance the implementation of voice-based health technologies into clinical practice and clinical research.

Abstract Current discussion surrounding the clinical capabilities of generative language models (GLMs) predominantly center around multiple-choice question-answer (MCQA) benchmarks derived from clinical licensing examinations. While accepted for human examinees, characteristics unique to GLMs bring into question the validity of such benchmarks. Here, we validate four benchmarks using eight GLMs, ablating for parameter size and reasoning capabilities, validating via prompt permutation three key assumptions that underpin the generalizability of MCQA-based assessments: that knowledge is applied, not memorized, that semantic consistency will lead to consistent answers, and that situations with no answers can be recognized. While large models are more resilient to our perturbations compared to small models, we globally invalidate these assumptions, with implications for reasoning models. Additionally, despite retaining the knowledge, small models are prone to answer memorization. All models exhibit significant failure in null-answer scenarios. We then suggest several adaptations for more robust benchmark designs more reflective of real-world conditions.

Abstract Objective Unlocking clinical information embedded in clinical notes has been hindered to a significant degree by domain-specific and context-sensitive language. Identification of note sections and structural document elements has been shown to improve information extraction and dependent downstream clinical natural language processing (NLP) tasks and applications. This study investigates the viability of a dynamic example selection prompting method to section classification using lightweight, open-source large language models (LLMs) as a practical solution for real-world healthcare clinical NLP systems. Materials and Methods We develop a dynamic few-shot prompting approach to classifying sections where section samples are first embedded using a transformer-based model and deposited in a vector store. During inference, the embedded samples with the most similar contextual embeddings to a given input section text are retrieved from the vector store and inserted into the LLM prompt. We evaluate this technique on two datasets comprising two section schemas, including varying levels of context. We compare the performance to baseline zero-shot and randomly selected few-shot scenarios. Results The dynamic few-shot prompting experiments yielded the highest F1 scores in each of the classification tasks and datasets for all seven of the LLMs included in the evaluation, averaging a macro F1 increase of 39.3% and 21.1% in our primary section classification task over the zero-shot and static few-shot baselines, respectively. Discussion and Conclusion Our results showcase substantial performance improvements imparted by dynamically selecting examples for few-shot LLM prompting, and further improvement by including section context, demonstrating compelling potential for clinical applications.

Abstract Objective Growing numbers of academic medical centers offer patient cohort discovery tools to their researchers, yet the performance of systems for this use case is not well understood. The objective of this research was to assess patient-level information retrieval methods using electronic health records for different types of cohort definition retrieval. Materials and Methods We developed a test collection consisting of about 100 000 patient records and 56 test topics that characterized patient cohort requests for various clinical studies. Automated information retrieval tasks using word-based approaches were performed, varying 4 different parameters for a total of 48 permutations, with performance measured using B-Pref. We subsequently created structured Boolean queries for the 56 topics for performance comparisons. In addition, we performed a more detailed analysis of 10 topics. Results The best-performing word-based automated query parameter settings achieved a mean B-Pref of 0.167 across all 56 topics. The way a topic was structured (topic representation) had the largest impact on performance. Performance not only varied widely across topics, but there was also a large variance in sensitivity to parameter settings across the topics. Structured queries generally performed better than automated queries on measures of recall and precision but were still not able to recall all relevant patients found by the automated queries. Conclusion While word-based automated methods of cohort retrieval offer an attractive solution to the labor-intensive nature of this task currently used at many medical centers, we generally found suboptimal performance in those approaches, with better performance obtained from structured Boolean queries. Future work will focus on using the test collection to develop and evaluate new approaches to query structure, weighting algorithms, and application of semantic methods.

AbstractObjectivesTo create test collections for evaluating clinical information retrieval (IR) systems and advancing clinical IR research.Materials and MethodsElectronic health record (EHR) data, including structured and free-text data, from 45 000 patients who are a part of the Mayo Clinic Biobank cohort was retrieved from the clinical data warehouse. The clinical IR system indexed a total of 42 million free-text EHR documents. The search queries consisted of 56 topics developed through a collaboration between Mayo Clinic and Oregon Health & Science University. We described the creation of test collections, including a to-be-evaluated document pool using five retrieval models, and human assessment guidelines. We analyzed the relevance judgment results in terms of human agreement and time spent, and results of three levels of relevance, and reported performance of five retrieval models.ResultsThe two judges had a moderate overall agreement with a Kappa value of 0.49, spent a consistent amount of time judging the relevance, and were able to identify easy and difficult topics. The conventional retrieval model performed best on most topics while a concept-based retrieval model had better performance on the topics requiring conceptual level retrieval.DiscussionIR can provide an alternate approach to leveraging clinical narratives for patient information discovery as it is less dependent on semantics. Our study showed the feasibility of test collections along with a few challenges.ConclusionThe conventional test collections for evaluating the IR system show potential for successfully evaluating clinical IR systems with a few challenges to be investigated.

Research in clinical information retrieval has long been stymied by the lack of open resources. However, both clinical information retrieval research innovation and legitimate privacy concerns can be served by the creation of intrainstitutional, fully protected resources. In this article, we provide some principles and tools for information retrieval resource‐building in the unique problem setting of patient‐level information retrieval, following the tradition of the Cranfield paradigm. We further include an analysis of parallel information retrieval resources at Oregon Health & Science University and Mayo Clinic that were built on these principles.