Collaço, Bernardo Gabriele

Background: Large language models (LLMs) are increasingly used in patient-facing clinical applications, creating a need for scalable methods to evaluate the accuracy, safety, and appropriateness of their responses. Although physician review remains the reference standard, it is resource-intensive and difficult to scale. Objective: To develop and clinically validate an LLM-as-a-judge framework for evaluating responses to patients’ periprocedural questions. Methods: We retrospectively evaluated 1285 patient–AIVA interactions from 112 patients across two Mayo Clinic sites and multiple surgical specialties. Physician reviewers classified interactions using a predefined true-positive (TP), false-negative (FN), true-negative (TN), and false-positive (FP) framework. The finalized LLM judge independently evaluated the same interactions. Agreement was assessed using four-class and category-specific agreement, Cohen’s kappa, and a secondary binary analysis of response correctness. Results: Overall, four-class agreement was 93.1% (95% CI, 91.7–94.5%), with Cohen’s κ = 0.852 (95% CI, 0.823–0.882). Category-specific agreement was 94.1% for TP, 90.7% for FN, 95.5% for TN, and 84.9% for FP classifications. In the binary correctness analysis, accuracy was 93.4% (95% CI, 92.0–94.7%), sensitivity 94.6%, specificity 90.2%, precision 96.2%, and F1-score 0.954. McNemar’s test demonstrated no significant asymmetry between paired classifications (p = 0.11). Conclusions: A clinically grounded LLM judge closely approximated physician evaluation of patient-facing AI responses. These findings support its potential as a scalable assistive monitoring tool while preserving physician oversight for uncertain or safety-sensitive interactions.

Background: Reliable access to postoperative guidance between clinic visits remains inconsistent, and the usefulness of large language models depends on how clinical knowledge is structured and retrieved. Knowledge graph-only (KG-only), vector retrieval-augmented generation (RAG-only), and hybrid graph-retrieval (GraphRAG) architectures have not been directly compared under controlled conditions for postoperative care. Objective: It was to compare these three pipelines on a curated postoperative question-answering benchmark using automated grounding metrics and blinded clinician review. Methods: A benchmark of 250 expert-authored questions (200 in-scope, 20 emergency-intent, 30 out-of-scope) was evaluated on an identical knowledge corpus using Gemini 2.5 Flash for generation and Gemini Embedding-001 for retrieval. Three blinded clinicians rated all 750 outputs; the primary outcome was majority-consensus accuracy across all 250 questions. GraphRAG is a multi-component pipeline that also includes query expansion, a second retrieval stage, evidence fusion, and a monotonic RAG fallback, and is therefore evaluated as a complete hybrid system rather than as graph retrieval in isolation. Results: Overall accuracy was 96.8% (95% CI: 93.8–98.4) for GraphRAG, 94.8% (95% CI: 91.3–96.9) for RAG-only, and 56.0% (95% CI: 49.8–62.0) for KG-only. Both retrieval-augmented systems outperformed KG-only (adjusted p < 0.001). The GraphRAG versus RAG-only difference was 2.0 percentage points, arose from five discordant question pairs and did not reach the Bonferroni-adjusted threshold (exact McNemar p = 0.062; adjusted p = 0.188). The monotonic RAG fallback determined 44.8% of GraphRAG outputs (112 of 250 queries). GraphRAG showed higher context recall (0.851 versus 0.796; adjusted p < 0.001), whereas RAG-only showed higher context precision. Estimated hallucination risk, derived from an LLM-as-a-judge procedure using Gemini 2.5 Flash rather than independent clinical adjudication, was 0.108, 0.132, and 0.436, respectively; the GraphRAG versus RAG-only difference was not significant (adjusted p = 0.771). Inter-rater agreement was high (Fleiss’ kappa = 0.854). Conclusions: Retrieval augmentation, rather than the choice of retrieval architecture, accounted for most of the difference observed. The incremental advantage of GraphRAG over a well-implemented RAG-only system was small, was not statistically significant after correction and reflects the complete hybrid system rather than graph augmentation alone. Emergency-intent accuracy was 75.0% for both retrieval-augmented systems; the architecture is therefore suited to supervised clinical workflows with clinician review, not autonomous patient-facing triage. Prospective validation on authentic patient messages and formal component ablations are required.

Surgeons are at high risk of developing long-term Work-Related Musculoskeletal Disorders (WMSDs) due to awkward sustained postures. Passive spine-supporting exoskeletons are a potential intervention to improve posture and reduce fatigue. The NekSpine ® was evaluated by 21 surgeons (15 male) across two hospital sites within a quaternary academic hospital system. Each surgeon performed at least two comparable procedures, one with the exoskeleton and one without. Surgeons completed pre- and postoperative surveys assessing NASA-TLX workload, usability, body-part discomfort, overall fatigue, and workflow disruptions. Eighty-two cases were evaluated (43 exoskeleton cases). Compared with baseline cases, surgeons reported reduced neck and upper back discomfort and lower overall fatigue when using the exoskeleton. The NekSpine ® exoskeleton may reduce spinal discomfort and fatigue during surgery, supporting its potential as an ergonomic intervention for surgeons.

Background: Agentic artificial intelligence (AI) systems, characterized by autonomous goal-directed behavior, multi-step reasoning, task decomposition, and tool use, are increasingly proposed for healthcare applications. However, their autonomy raises concerns regarding transparency, accountability, and human oversight. While explainable AI (XAI) has been widely studied in traditional predictive models, less is known about how explainability is implemented within agentic architectures. Objective: To map the emerging literature on explainable agentic AI (XAAI) in healthcare and characterize the types, scope, and forms of explainability used in these systems. Methods: A scoping review was conducted following PRISMA-ScR guidelines. PubMed, Embase, IEEE Xplore, and ACM Digital Library were searched through November 2025. Eligible studies described healthcare-related agentic AI systems incorporating explicit explainability mechanisms. Data were extracted on system architecture, explainability type (intrinsic, post hoc, hybrid), explanation scope (local, global), explanation form, and reported clinical outcomes. Results: Nine studies met the inclusion criteria. All systems demonstrated core agentic features, including autonomy, task decomposition, and tool integration, often within multi-agent frameworks. Explainability was predominantly intrinsic and workflow-native, typically delivered through textual reasoning traces and example-based grounding in retrieved clinical evidence. Feature-based and global explanations were comparatively rare and largely confined to hybrid architectures. Across domains including radiology, neurology, psychiatry, and biomedical research, XAAI systems were reported to improve performance and interpretability relative to baseline models in the included studies. However, these findings were derived from heterogeneous, predominantly experimental or retrospective studies, and structured human-in-the-loop oversight was infrequently described. Conclusions: Current XAAI systems appear to emphasize process transparency and evidence grounding rather than mechanistic model-level attribution. The available evidence remains limited and heterogeneous, and findings should be interpreted as early trends rather than established characteristics. Further progress will require standardized evaluation frameworks, clearer reporting of oversight mechanisms, and validation in real-world clinical settings to support safe and trustworthy integration of agentic AI into healthcare practice.

Background: Large language models (LLMs) are being used by surgeons for education and reference yet concerns about hallucinations and reliability limit safe adoption. Retrieval-augmented generation (RAG) can offer a potential solution by grounding responses in a high-quality external database (e.g., medical textbooks) to enhance accuracy. However, performance tradeoffs across different RAG configurations—many of which exponentially increase computational cost—remain poorly characterized. Methods: In total, 120 lightweight, open-source RAG configurations were evaluated across 40 plastic surgery-focused question-answering tasks (20 single-hop, 20 multi-hop), spanning multiple subspecialties (4800 total evaluations). Configurations varied by base LLM (Phi-3-mini-128k-instruct vs. BioMistral-7B), embedding model, database size, chunk size, and query hop type. Performance was assessed using semantic similarity (Ragas) to physician-validated reference answers. Performance was analyzed using linear mixed-effects regression with query as a random effect and fixed and interaction effects selected via likelihood testing and AIC. Results: High performance was achievable using lightweight, open-source models. While BioMistral-7B had high mean sematic similarity under specific configurations (mean semantic similarity up to 0.786), Phi-3-mini-128k-instruct demonstrated more consistent performance across query complexity. Larger database sizes significantly improved semantic similarity, with the largest gain at intermediate sizes (e.g., size 5: +0.043, p = 0.001). Embedding choice had a strong effect, with bge-large-en-v1.5 improving performance (p = 0.0016) and Bio_ClinicalBERT markedly reducing it (p < 0.001). Multi-hop queries substantially reduced performance (p < 0.001), though this effect was attenuated for Phi-3-mini-128k-instruct via a strong model × hop-type interaction (p < 0.001). Conclusions: RAG systems for plastic surgery do not require large proprietary models, as performance depends on configuration choices and interaction effects rather than isolated components. With advancements, predictive modeling may enable resource-efficient, safe deployment of clinical RAG systems.

Background: Automated discharge instructions are increasingly used to support post-discharge communication, patient education, and nursing follow-up, yet the current state remains unidentified. This systematic review explores the types of automated discharge instructions used and their effectiveness in enhancing patient engagement and reducing readmission, emergency department visits and reoperation rates. Methods: A systematic search was conducted on 15 April 2025, using Embase, PubMed, Scopus, Web of Science, and CINAHL, following PRISMA guidelines. Inclusion criteria required peer-reviewed original research evaluating the utilization of automated patient discharge instructions following hospital admission or surgical stay. Exclusion criteria included correspondence, reviews, educational materials, not peer-reviewed, retracted reports, not retrievable, and no English translation. Risk of bias was assessed independently using NIH, JBI, ROB-2, and ROBINS-I tools. Two investigators independently conducted the screening, extraction, and synthesis of results using Endnote and Microsoft Excel. Results: Of the 1252 records identified, 13 studies were selected for analysis. There was a total of 34,386 patients across a diverse range of healthcare settings and clinical contexts. The average sample size per study was approximately 4912, with study samples ranging from 16 to 13,188 patients. The modalities of discharge instructions included automated phone calls (23.1%) and/or text messages (53.8%), as well as printed out auto-generated summaries (15.4%). Patient engagement was generally high, with automated phone calls showing the most consistent interaction, with completion rates ranging from 44% to 56%, often prompting clinical follow-up. SMS tools demonstrated strong scalability and response rates up to 87%. Two studies reported on hospital readmission outcomes and only a single study reported on emergency department revisit rates, while none assessed reoperation outcomes. Among those reporting readmission, automated phone calls and SMS were associated with lower or proxy-reduced readmission rates. Included studies had low to moderate levels of bias. Conclusions: While evidence on clinical outcomes such as readmissions, emergency department revisits, and reoperations remains limited and inconclusive, automated discharge tools—particularly phone calls and SMS—consistently demonstrated high patient engagement. Automated discharge tools show promise for supporting transitional care, discharge education, and post-discharge monitoring, highlighting the future role of automated tools in nursing workflows to support follow-up, escalation, and continuity of care.

(1) Background: Large language models (LLMs) show promise in healthcare but are constrained by hallucinations, static knowledge, and limited domain specificity. Fine-tuning (FT) and retrieval-augmented generation (RAG) offer complementary solutions, with FT embedding domain reasoning and RAG enabling dynamic, up-to-date knowledge access. Hybrid FT + RAG frameworks have been proposed to improve factual accuracy and clinical reliability. This scoping review synthesizes current evidence on such hybrids in healthcare AI. (2) Methods: The search across PubMed, IEEE Xplore, Google Scholar, and Embase identified studies implementing explicit FT + RAG hybrids in healthcare or biomedical tasks. Eligible studies reported empirical evaluations of LLM performance or behavior. Data were extracted on base models, FT strategies, RAG architectures, applications, and performance outcomes. (3) Results: Seven studies met inclusion criteria. FT + RAG systems consistently outperformed FT-only or RAG-only approaches across QA, clinical summarization, report generation, and decision support tasks. Parameter-efficient FT methods (e.g., LoRA) were common, while RAG implementations varied (dense, hybrid, hierarchical, multimodal, federated). Reported benefits included improved accuracy, reduced hallucination, and greater clinician preference and feasibility in protected settings. (4) Conclusions: FT + RAG frameworks represent a promising direction for clinically grounded healthcare AI, combining domain-specific reasoning with transparent, up-to-date retrieval. Future work should prioritize standardized evaluation, workflow integration, and governance to enable safe deployment.

Background: Large language models (LLMs) are increasingly integrated into clinical documentation, decision support, and patient-facing applications across healthcare, including plastic and reconstructive surgery. Yet, their evaluation remains bottlenecked by costly, time-consuming human review. This has given rise to LLM-as-a-judge, in which LLMs are used to evaluate the outputs of other AI systems. Methods: This review examines LLM-as-a-judge in healthcare with particular attention to judging architectures, validation strategies, and emerging applications. A narrative review of the literature was conducted, synthesizing LLM judge methodologies as well as judging paradigms, including those applied to clinical documentation, medical question-answering systems, and clinical conversation assessment. Results: Across tasks, LLM judges align most closely with clinicians on objective criteria (e.g., factuality, grammaticality, internal consistency), benefit from structured evaluation and chain-of-thought prompting, and can approach or exceed inter-clinician agreement, but remain limited for subjective or affective judgments and by dataset quality and task specificity. Conclusions: The literature indicates that LLM judges can enable efficient, standardized evaluation in controlled settings; however, their appropriate role remains supportive rather than substitutive, and their performance may not generalize to complex plastic surgery environments. Their safe use depends on rigorous human oversight and explicit governance structures.

Background: Generative AI and synthetic media have enabled realistic human Embodied Conversational Agents (ECAs) or avatars. A subset of this technology replicates faces and voices to create realistic likenesses. When combined with avatars, these methods enable the creation of “digital twins” of physicians, offering patients scalable, 24/7 clinical communication outside the immediate clinical environment. This study evaluated surgical patient perceptions of an AI-generated surgeon avatar for postoperative education. Methods: We conducted a pilot feasibility study with 30 plastic surgery patients at Mayo Clinic, USA (July–August 2025). A bespoke interactive surgeon avatar was developed in Python using the HeyGen IV model to reproduce the surgeon’s likeness. Patients interacted with the avatar through natural voice queries, which were mapped to predetermined, pre-recorded video responses covering ten common postoperative topics. Patient perceptions were assessed using validated scales of usability, engagement, trust, eeriness, and realism, supplemented by qualitative feedback. Results: The avatar system reliably answered 297 of 300 patient queries (99%). Usability was excellent (mean System Usability Scale score = 87.7 ± 11.5) and engagement high (mean 4.27 ± 0.23). Trust was the highest-rated domain, with all participants (100%) finding the avatar trustworthy and its information believable. Eeriness was minimal (mean = 1.57 ± 0.48), and 96.7% found the avatar visually pleasing. Most participants (86.6%) recognized the avatar as their surgeon, although many still identified it as artificial; voice resemblance was less convincing (70%). Interestingly, participants with prior exposure to deepfakes demonstrated consistently higher acceptance, rating usability, trust, and engagement 5–10% higher than those without prior exposure. Qualitative feedback highlighted clarity, efficiency, and convenience, while noting limitations in realism and conversational scope. Conclusions: The AI-generated physician avatar achieved high patient acceptance without triggering uncanny valley effects. Transparency about the synthetic nature of the technology enhanced, rather than diminished, trust. Familiarity with the physician and institutional credibility likely played a key role in the high trust scores observed. When implemented transparently and with appropriate safeguards, synthetic physician avatars may offer a scalable solution for postoperative education while preserving trust in clinical relationships.

Abstract Background Outcomes of patients with acute pyelonephritis (AP) treated in a hospital-at-home setting have not been comprehensively evaluated in the United States. Methods We performed a multicenter, retrospective cohort study of adults diagnosed with and managed for AP in Mayo Clinic's Advanced Care at Home (ACH) program between July 2020 and January 2025. Collected data included demographics, Charlson Comorbidity Index (CCI), genitourinary comorbidities, severity of illness (SOI), and risk of mortality (ROM) scores, as well as pyelonephritis-related complications. Outcomes included length of stay (LOS), escalation of care, and 30-day postdischarge emergency department (ED) visits, readmissions, and mortality. Results A total of 165 patients met inclusion criteria. Median age was 67 years; SOI scores were moderate in 33.3%, major in 52.1%, and extreme in 8.5%. ROM scores were moderate in 30.3%, major in 38.2%, and extreme in 6.7%. Median CCI score was 5, and all patients had preexisting genitourinary conditions. On admission, 30.9% met sepsis criteria, acute kidney injury was present in 47.3%, and bacteremia developed in 33.3%. Median LOS in the ACH program was 3.1 days. Only 4.8% required escalation to a brick-and-mortar hospital. Readmission occurred in 17.0%, and 4.8% had ED visits. No in-program deaths occurred. Conclusions This multicenter retrospective study shows that AP, including cases with high illness severity and complex comorbidities, can be managed safely and effectively in a hospital-at-home setting with careful patient selection and robust infrastructure to support timely escalation when needed.