Liu, Hongfang

Abstract Background Integrating advanced artificial intelligence (AI) into clinical decision-support often requires the sharing of sensitive patient data with external services, raising privacy concerns. Homomorphic encryption (HE) allows computing directly on encrypted data, without revealing the underlying patient information. Objectives To develop a large language model (LLM)-assisted diagnosis framework while preserving patient privacy in the clinical text analysis, by leveraging HE and using rare disease (RD) diagnosis as a representative application. To demonstrate HE does not hinder the system performance. Materials and Methods Texts from patient histories and a RD knowledge base were embedded by LLMs into vectors, then encrypted using HE to obscure private information while retaining the semantic nuances. Diagnostic recommendations were generated by computing and ranking the similarities between the patient history and RD vectors in the encrypted space. The system was evaluated using 50 synthetic case reports (5 RDs, each with 10 reports). Results Applying HE did protect private information from reverse-embedding attacks. HE imposed little disruption to the diagnostic accuracy, with normalized discounted cumulative gains (nDCG) of 0.6108 ± 0.3412 (encrypted) versus 0.6083 ± 0.3415 (unencrypted). The accuracy and computational performance were tunable and consistent, as demonstrated across five different LLMs. Discussion Our privacy-preserving framework opens tremendous opportunities toward hosting and serving powerful AI solutions across institution boundaries, which would remove the need for local deidentification and incentivize users to access secure external decision-support services. Conclusions Integrating HE with LLM retrieval can promote the dissemination of nonredundant, high-capacity AI services by preserving both privacy and accuracy.

Abstract Background Delirium is often underdiagnosed in clinical practice and is not routinely coded for billing. While manual chart review can identify delirium, it is labor-intensive and impractical for large-scale studies. Natural language processing (NLP) can analyze unstructured text in electronic health records (EHRs) to extract meaningful clinical information. Methods To support national integration of NLP for EHR-based delirium identification across different institutions, we launched the Delirium Interest Group within the national Evolve to Next-Gen Accrual to Clinical Trials (ENACT) NLP Working Group. This paper outlines our initial efforts to standardize, evaluate, and translate an NLP-based delirium detection model into the i2b2/ENACT platform. Results Multisite contextual inquiry identified several key challenges, including variations in local screening practices (eg, tools used, documentation frequency, and quality control), the need for harmonized definitions in the context of EHRs, and the complexity of modeling temporal logic. Multisite NLP evaluation revealed variable performance degradation driven by differences in delirium screening practices, clinical documentation patterns and semantics, and note syntactic structures. Conclusion Our work represents an important first step toward enabling scalable and standardized NLP-based delirium detection across institutions. By engaging diverse institutions through the ENACT NLP Working Group, we identified shared challenges and site-specific variations that impact model implementation and performance. Our collaborative approach enabled the development of a more robust framework for delirium identification across heterogeneous EHR systems. Future efforts will build on this foundation to enhance the validity, usability, and translational impact of delirium detection.

Abstract Error analysis is a critical step in evaluating and improving clinical concept extraction models, the most common clinical natural language processing (NLP) task. Unlike corpus annotation, which follows standardized protocols for creating gold-standard datasets, error analysis requires nuanced judgment grounded in both clinical expertise and NLP knowledge. This is especially important given the heterogeneity of clinical text, where variations in documentation style, note structure, and terminology can substantially influence model behavior. Despite its importance, there is currently no standardized, user-level framework to support systematic error analysis in clinical concept extraction task. In this study, we developed and validated MedError, a machine-assisted, human-in-the-loop framework designed to standardize and enhance error analysis for clinical concept extraction tasks. We collected and manually curated a corpus of 1,187 unique errors from a total of 4,237 notes across three different distinct hospitals. The error categories were defined using our previously validated error taxonomy and included 480 false negatives and 707 false positives across 25 error types and 48 clinical concept categories. We evaluated the performance of three proprietary and three open-source large language models (LLMs) in automatically classifying these errors into 26 and 15 predefined categories. We further developed a machine-assisted framework, MedError, which integrates best practices in error analysis, LLM-assisted classification and reasoning, and a user-friendly interface to enable more efficient, reproducible, and context-aware error analysis. The framework supports both single-site and federated multisite error analysis, facilitating the effective translation of clinical NLP systems into real-world settings.

Abstract Objective: Electronic Health Record (EHR) data are critical for advancing translational research and AI technologies. The ENACT network offers access to structured EHR data across 57 CTSA hubs. However, substantial information is contained in clinical narratives, requiring natural language processing (NLP) for research. The ENACT NLP Working Group was formed to make NLP-derived clinical information accessible and queryable across the network. Methods: We established the ENACT NLP Working Group with 13 sites selected based on criteria including clinical notes access, IT infrastructure, NLP expertise, and institutional support. We divided sites into five focus groups targeting clinical tasks within disease contexts. Each focus group consisted of two development sites and two validation sites. We extended the ENACT ontology to standardize NLP-derived data and conducted multisite evaluations using the Open Health Natural Language Processing (OHNLP) Toolkit. Results: The working group achieved 100% site retention and deployed NLP infrastructure across all sites. We developed and validated NLP algorithms for rare disease phenotyping, social determinants of health, opioid use disorder, sleep phenotyping, and delirium phenotyping. Performance varied across sites (F1 scores 0.53–0.96), highlighting data heterogeneity impacts. We extended the ENACT common data model and ontology to incorporate NLP-derived data while maintaining Shared Health Research Informatics NEtwork (SHRINE) compatibility. Conclusion: This demonstrates feasibility of deploying NLP infrastructure across large, federated networks. The focus group approach proved more practical than general-purpose approaches. Key lessons include the challenge of data heterogeneity and importance of collaborative governance. This work also provides a foundation that other networks can build on to implement NLP capabilities for translational research.

BACKGROUND: The impact of covert cerebrovascular disease on falls in the general population is not well-known. Here, we determine the time to a first fall following incidentally detected covert cerebrovascular disease during a clinical neuroimaging episode. METHODS: This longitudinal cohort study assessed computed tomography (CT) and magnetic resonance imaging from 2009 to 2019 of patients aged >50 years registered with Kaiser Permanente Southern California which is a healthcare organization combining health plan coverage with coordinated medical services, excluding those with before stroke/dementia. We extracted evidence of incidental covert brain infarcts (CBI) and white matter hyperintensities/hypoattenuation (WMH) from imaging reports using natural language processing. We examined associations of CBI and WMH with falls requiring medical attention, using Cox proportional hazards regression models with adjustment for 12 variables including age, sex, ethnicity multimorbidity, polypharmacy, and incontinence. RESULTS: We assessed 241 050 patients, mean age 64.9 (SD, 10.42) years, 61.3% female, detecting covert cerebrovascular disease in 31.1% over a mean follow-up duration of 3.04 years. A recorded fall occurred in 21.2% (51 239/241 050) during follow-up. On CT, single fall incidence rate/1000 person-years (p-y) was highest in individuals with both CBI and WMH on CT (129.3 falls/1000 p-y [95% CI, 123.4–135.5]), followed by WMH (109.9 falls/1000 p-y [108.0–111.9]). On magnetic resonance imaging, the incidence rate was the highest with both CBI and WMH (76.3 falls/1000 p-y [95% CI, 69.7–83.2]), followed by CBI (71.4 falls/1000 p-y [95% CI, 65.9–77.2]). The adjusted hazard ratio for single index fall in individuals with CBI on CT was 1.13 (95% CI, 1.09–1.17); versus magnetic resonance imaging 1.17 (95% CI, 1.08–1.27). On CT, the risk for single index fall incrementally increased for mild (1.37 [95% CI, 1.32–1.43]), moderate (1.57 [95% CI, 1.48–1.67]), or severe WMH (1.57 [95% CI, 1.45–1.70]). On magnetic resonance imaging, index fall risk similarly increased with increasing WMH severity: mild (1.11 [95% CI, 1.07–1.17]), moderate (1.21 [95% CI, 1.13–1.28]), and severe WMH (1.34 [95% CI, 1.22–1.46]). CONCLUSIONS: In a large population with neuroimaging, CBI and WMH are independently associated with greater risks of an index fall. Increasing severities of WMH are associated incrementally with fall risk across imaging modalities.

Background Electronic health records (EHRs) serve as a critical tool for clinical documentation, yet their complexity has raised concerns about readability and usability for healthcare providers and patients. Readability scores, such as the Flesch-Kincaid (FK) Grade Level, offer a quantitative method to assess documentation clarity. In this study, the authors evaluated the impact of an EHR transition on the readability of clinical notes across multiple medical specialties. Methods All clinical notes recorded between 2016 and 2019 at a large healthcare organization were evaluated across specialties. The organization transitioned EHR systems during the study period, in May 2018. FK Grade Level scores were calculated for each clinical consultation note and stratified by specialty and EHR. Results Across 10 medical specialties, 630,246 clinical notes were evaluated. The median FK grade levels from notes composed in the new EHR were similar to those composed in the previous EHR (8.228 vs 8.240; p=0.58). The variation in FK grade level as measured by standard deviation was lower in the new EHR compared with the old EHR (1.405 vs 1.764; p <0.01). Conclusions Calculation of FK grade level of clinical notes is a feasible method to assess clinical note readability. Study results showed that clinical documentation readability may not be strongly associated with underlying EHR. Migrating to a new EHR alone should not be expected to improve readability of clinical notes.

Abstract Objective Unlocking clinical information embedded in clinical notes has been hindered to a significant degree by domain-specific and context-sensitive language. Identification of note sections and structural document elements has been shown to improve information extraction and dependent downstream clinical natural language processing (NLP) tasks and applications. This study investigates the viability of a dynamic example selection prompting method to section classification using lightweight, open-source large language models (LLMs) as a practical solution for real-world healthcare clinical NLP systems. Materials and Methods We develop a dynamic few-shot prompting approach to classifying sections where section samples are first embedded using a transformer-based model and deposited in a vector store. During inference, the embedded samples with the most similar contextual embeddings to a given input section text are retrieved from the vector store and inserted into the LLM prompt. We evaluate this technique on two datasets comprising two section schemas, including varying levels of context. We compare the performance to baseline zero-shot and randomly selected few-shot scenarios. Results The dynamic few-shot prompting experiments yielded the highest F1 scores in each of the classification tasks and datasets for all seven of the LLMs included in the evaluation, averaging a macro F1 increase of 39.3% and 21.1% in our primary section classification task over the zero-shot and static few-shot baselines, respectively. Discussion and Conclusion Our results showcase substantial performance improvements imparted by dynamically selecting examples for few-shot LLM prompting, and further improvement by including section context, demonstrating compelling potential for clinical applications.

ASCO Guidelines provide recommendations with comprehensive review and analyses of the relevant literature for each recommendation, following the guideline development process as outlined in the ASCO Guidelines Methodology Manual . ASCO Guidelines follow the ASCO Conflict of Interest Policy for Clinical Practice Guidelines . Clinical Practice Guidelines and other guidance (“Guidance”) provided by ASCO is not a comprehensive or definitive guide to treatment options. It is intended for voluntary use by clinicians and should be used in conjunction with independent professional judgment. Guidance may not be applicable to all patients, interventions, diseases or stages of diseases. Guidance is based on review and analysis of relevant literature, and is not intended as a statement of the standard of care. ASCO does not endorse third-party drugs, devices, services, or therapies and assumes no responsibility for any harm arising from or related to the use of this information. See complete disclaimer in Appendix 1 and 2 (online only) for more . PURPOSE To provide evidence-based recommendations for patients with metastatic castration-resistant prostate cancer (mCRPC). METHODS An Expert Panel including patient representation completed a systematic review of the evidence and made recommendations. RESULTS Depending upon prior treatment received, androgen receptor pathway inhibitors (ARPIs: enzalutamide, abiraterone with prednisone), poly(ADP-ribose) polymerase inhibitors (PARPi), chemotherapeutic agents (docetaxel, cabazitaxel), radiopharmaceuticals (radium 223, 177 Lu-prostate-specific membrane antigen [PSMA]-617), and sipuleucel-T have demonstrated an overall survival (OS) benefit for patients with mCRPC. For patients with BRCA1 / 2 alterations who did not receive prior ARPI, the combination of PARPi and ARPI (talazoparib + enzalutamide, olaparib and/or niraparib + abiraterone) has shown clinical benefit. For patients with BRCA1 / 2 alterations who received prior ARPI or ARPI followed by docetaxel, olaparib showed OS benefit. In select patients with microsatellite instability-high/mismatch repair-deficient, pembrolizumab showed clinical efficacy. RECOMMENDATIONS Prior systemic therapy for castration-sensitive prostate cancer will determine subsequent therapy used for mCRPC. Continue androgen-deprivation therapy for patients with mCRPC indefinitely. Early adoption of somatic genetic testing and palliative care is recommended. Patients with mCRPC and bony metastases should receive a bone-protective agent. The panel recommends the combination of ARPI with PARPi in patients with BRCA1 / 2 alterations who did not receive prior ARPI. For patients who received prior ARPI, the panel recommends docetaxel chemotherapy. The panel recommends 177 Lu-PSMA-617 or cabazitaxel chemotherapy for patients who receive prior ARPI and docetaxel chemotherapy. For patients with BRCA1 / 2 alterations who received prior ARPI, the panel recommends PARPi monotherapy. Radium 223 is recommended for patients with symptomatic bone–only disease. Evidence for optimal sequencing for mCRPC regimens is lacking. Additional information is available at www.asco.org/genitourinary-cancer-guidelines .

Introduction While incidentally discovered covert cerebrovascular diseases (id-CCD) are associated with future stroke, it is not known if patients with id-CCD are prescribed statins. Methods Patients age ≥50 with id-CCD on neuroimaging from 2009 to 2019 with no prior ischaemic stroke, transient ischaemic attack or dementia were identified using natural language processing in a large real-world cohort. Robust Poisson multivariable regression was used to assess statin prescription among patients without prior statins. Results Among 2 41 050 patients, 74 975 patients (31.1%; 4.7% with covert brain infarcts (CBI); 29.0% with white matter disease (WMD)) had id-CCD. 53.5% (95% CI 53.2 to 53.9%) were not on statins within 6 months prior to the scan. Of those, 12.0% (95% CI 11.7 to 12.3%) were prescribed statins in the next 6 months compared with 9.3% (95% CI 9.1 to 9.4%) in those without CCD, a 2.7% (95% CI 2.4 to 3.1%) absolute increase in statin prescription for those with id-CCD. In adjusted analyses, the presence of id-CCD was only associated with minor increases in statin prescription (CBI or WMD (risk ratio (RR) 1.09, 95% CI 1.05 to 1.13), CBI alone (RR 1.34, 95% CI 1.21 to 1.47), WMD alone (RR 1.05, 95% CI 1.01 to 1.09), and CBI and WMD (RR 1.23, 95% CI 1.12 to 1.35)). Discussion Identification of id-CCD is not associated with substantial changes in statin prescription in routine clinical practice.

Abstract Objectives The surge in patient portal messages (PPMs) with increasing needs and workloads for efficient PPM triage in healthcare settings has spurred the exploration of AI-driven solutions to streamline the healthcare workflow processes, ensuring timely responses to patients to satisfy their healthcare needs. However, there has been less focus on isolating and understanding patient primary concerns in PPMs—a practice which holds the potential to yield more nuanced insights and enhances the quality of healthcare delivery and patient-centered care. Materials and Methods We propose a fusion framework to leverage pretrained language models (LMs) with different language advantages via a Convolution Neural Network for precise identification of patient primary concerns via multi-class classification. We examined 3 traditional machine learning models, 9 BERT-based language models, 6 fusion models, and 2 ensemble models. Results The outcomes of our experimentation underscore the superior performance achieved by BERT-based models in comparison to traditional machine learning models. Remarkably, our fusion model emerges as the top-performing solution, delivering a notably improved accuracy score of 77.67 ± 2.74% and an F1 score of 74.37 ± 3.70% in macro-average. Discussion This study highlights the feasibility and effectiveness of multi-class classification for patient primary concern detection and the proposed fusion framework for enhancing primary concern detection. Conclusions The use of multi-class classification enhanced by a fusion of multiple pretrained LMs not only improves the accuracy and efficiency of patient primary concern identification in PPMs but also aids in managing the rising volume of PPMs in healthcare, ensuring critical patient communications are addressed promptly and accurately.