Forte, Antonio Jorge

Background: Large language models (LLMs) are being used by surgeons for education and reference yet concerns about hallucinations and reliability limit safe adoption. Retrieval-augmented generation (RAG) can offer a potential solution by grounding responses in a high-quality external database (e.g., medical textbooks) to enhance accuracy. However, performance tradeoffs across different RAG configurations—many of which exponentially increase computational cost—remain poorly characterized. Methods: In total, 120 lightweight, open-source RAG configurations were evaluated across 40 plastic surgery-focused question-answering tasks (20 single-hop, 20 multi-hop), spanning multiple subspecialties (4800 total evaluations). Configurations varied by base LLM (Phi-3-mini-128k-instruct vs. BioMistral-7B), embedding model, database size, chunk size, and query hop type. Performance was assessed using semantic similarity (Ragas) to physician-validated reference answers. Performance was analyzed using linear mixed-effects regression with query as a random effect and fixed and interaction effects selected via likelihood testing and AIC. Results: High performance was achievable using lightweight, open-source models. While BioMistral-7B had high mean sematic similarity under specific configurations (mean semantic similarity up to 0.786), Phi-3-mini-128k-instruct demonstrated more consistent performance across query complexity. Larger database sizes significantly improved semantic similarity, with the largest gain at intermediate sizes (e.g., size 5: +0.043, p = 0.001). Embedding choice had a strong effect, with bge-large-en-v1.5 improving performance (p = 0.0016) and Bio_ClinicalBERT markedly reducing it (p < 0.001). Multi-hop queries substantially reduced performance (p < 0.001), though this effect was attenuated for Phi-3-mini-128k-instruct via a strong model × hop-type interaction (p < 0.001). Conclusions: RAG systems for plastic surgery do not require large proprietary models, as performance depends on configuration choices and interaction effects rather than isolated components. With advancements, predictive modeling may enable resource-efficient, safe deployment of clinical RAG systems.

(1) Background: Large language models (LLMs) show promise in healthcare but are constrained by hallucinations, static knowledge, and limited domain specificity. Fine-tuning (FT) and retrieval-augmented generation (RAG) offer complementary solutions, with FT embedding domain reasoning and RAG enabling dynamic, up-to-date knowledge access. Hybrid FT + RAG frameworks have been proposed to improve factual accuracy and clinical reliability. This scoping review synthesizes current evidence on such hybrids in healthcare AI. (2) Methods: The search across PubMed, IEEE Xplore, Google Scholar, and Embase identified studies implementing explicit FT + RAG hybrids in healthcare or biomedical tasks. Eligible studies reported empirical evaluations of LLM performance or behavior. Data were extracted on base models, FT strategies, RAG architectures, applications, and performance outcomes. (3) Results: Seven studies met inclusion criteria. FT + RAG systems consistently outperformed FT-only or RAG-only approaches across QA, clinical summarization, report generation, and decision support tasks. Parameter-efficient FT methods (e.g., LoRA) were common, while RAG implementations varied (dense, hybrid, hierarchical, multimodal, federated). Reported benefits included improved accuracy, reduced hallucination, and greater clinician preference and feasibility in protected settings. (4) Conclusions: FT + RAG frameworks represent a promising direction for clinically grounded healthcare AI, combining domain-specific reasoning with transparent, up-to-date retrieval. Future work should prioritize standardized evaluation, workflow integration, and governance to enable safe deployment.

Background: Large language models (LLMs) are increasingly integrated into clinical documentation, decision support, and patient-facing applications across healthcare, including plastic and reconstructive surgery. Yet, their evaluation remains bottlenecked by costly, time-consuming human review. This has given rise to LLM-as-a-judge, in which LLMs are used to evaluate the outputs of other AI systems. Methods: This review examines LLM-as-a-judge in healthcare with particular attention to judging architectures, validation strategies, and emerging applications. A narrative review of the literature was conducted, synthesizing LLM judge methodologies as well as judging paradigms, including those applied to clinical documentation, medical question-answering systems, and clinical conversation assessment. Results: Across tasks, LLM judges align most closely with clinicians on objective criteria (e.g., factuality, grammaticality, internal consistency), benefit from structured evaluation and chain-of-thought prompting, and can approach or exceed inter-clinician agreement, but remain limited for subjective or affective judgments and by dataset quality and task specificity. Conclusions: The literature indicates that LLM judges can enable efficient, standardized evaluation in controlled settings; however, their appropriate role remains supportive rather than substitutive, and their performance may not generalize to complex plastic surgery environments. Their safe use depends on rigorous human oversight and explicit governance structures.