Liu, Hongfang

Abstract Background Delirium is often underdiagnosed in clinical practice and is not routinely coded for billing. While manual chart review can identify delirium, it is labor-intensive and impractical for large-scale studies. Natural language processing (NLP) can analyze unstructured text in electronic health records (EHRs) to extract meaningful clinical information. Methods To support national integration of NLP for EHR-based delirium identification across different institutions, we launched the Delirium Interest Group within the national Evolve to Next-Gen Accrual to Clinical Trials (ENACT) NLP Working Group. This paper outlines our initial efforts to standardize, evaluate, and translate an NLP-based delirium detection model into the i2b2/ENACT platform. Results Multisite contextual inquiry identified several key challenges, including variations in local screening practices (eg, tools used, documentation frequency, and quality control), the need for harmonized definitions in the context of EHRs, and the complexity of modeling temporal logic. Multisite NLP evaluation revealed variable performance degradation driven by differences in delirium screening practices, clinical documentation patterns and semantics, and note syntactic structures. Conclusion Our work represents an important first step toward enabling scalable and standardized NLP-based delirium detection across institutions. By engaging diverse institutions through the ENACT NLP Working Group, we identified shared challenges and site-specific variations that impact model implementation and performance. Our collaborative approach enabled the development of a more robust framework for delirium identification across heterogeneous EHR systems. Future efforts will build on this foundation to enhance the validity, usability, and translational impact of delirium detection.

Abstract Error analysis is a critical step in evaluating and improving clinical concept extraction models, the most common clinical natural language processing (NLP) task. Unlike corpus annotation, which follows standardized protocols for creating gold-standard datasets, error analysis requires nuanced judgment grounded in both clinical expertise and NLP knowledge. This is especially important given the heterogeneity of clinical text, where variations in documentation style, note structure, and terminology can substantially influence model behavior. Despite its importance, there is currently no standardized, user-level framework to support systematic error analysis in clinical concept extraction task. In this study, we developed and validated MedError, a machine-assisted, human-in-the-loop framework designed to standardize and enhance error analysis for clinical concept extraction tasks. We collected and manually curated a corpus of 1,187 unique errors from a total of 4,237 notes across three different distinct hospitals. The error categories were defined using our previously validated error taxonomy and included 480 false negatives and 707 false positives across 25 error types and 48 clinical concept categories. We evaluated the performance of three proprietary and three open-source large language models (LLMs) in automatically classifying these errors into 26 and 15 predefined categories. We further developed a machine-assisted framework, MedError, which integrates best practices in error analysis, LLM-assisted classification and reasoning, and a user-friendly interface to enable more efficient, reproducible, and context-aware error analysis. The framework supports both single-site and federated multisite error analysis, facilitating the effective translation of clinical NLP systems into real-world settings.

BACKGROUND: The impact of covert cerebrovascular disease on falls in the general population is not well-known. Here, we determine the time to a first fall following incidentally detected covert cerebrovascular disease during a clinical neuroimaging episode. METHODS: This longitudinal cohort study assessed computed tomography (CT) and magnetic resonance imaging from 2009 to 2019 of patients aged >50 years registered with Kaiser Permanente Southern California which is a healthcare organization combining health plan coverage with coordinated medical services, excluding those with before stroke/dementia. We extracted evidence of incidental covert brain infarcts (CBI) and white matter hyperintensities/hypoattenuation (WMH) from imaging reports using natural language processing. We examined associations of CBI and WMH with falls requiring medical attention, using Cox proportional hazards regression models with adjustment for 12 variables including age, sex, ethnicity multimorbidity, polypharmacy, and incontinence. RESULTS: We assessed 241 050 patients, mean age 64.9 (SD, 10.42) years, 61.3% female, detecting covert cerebrovascular disease in 31.1% over a mean follow-up duration of 3.04 years. A recorded fall occurred in 21.2% (51 239/241 050) during follow-up. On CT, single fall incidence rate/1000 person-years (p-y) was highest in individuals with both CBI and WMH on CT (129.3 falls/1000 p-y [95% CI, 123.4–135.5]), followed by WMH (109.9 falls/1000 p-y [108.0–111.9]). On magnetic resonance imaging, the incidence rate was the highest with both CBI and WMH (76.3 falls/1000 p-y [95% CI, 69.7–83.2]), followed by CBI (71.4 falls/1000 p-y [95% CI, 65.9–77.2]). The adjusted hazard ratio for single index fall in individuals with CBI on CT was 1.13 (95% CI, 1.09–1.17); versus magnetic resonance imaging 1.17 (95% CI, 1.08–1.27). On CT, the risk for single index fall incrementally increased for mild (1.37 [95% CI, 1.32–1.43]), moderate (1.57 [95% CI, 1.48–1.67]), or severe WMH (1.57 [95% CI, 1.45–1.70]). On magnetic resonance imaging, index fall risk similarly increased with increasing WMH severity: mild (1.11 [95% CI, 1.07–1.17]), moderate (1.21 [95% CI, 1.13–1.28]), and severe WMH (1.34 [95% CI, 1.22–1.46]). CONCLUSIONS: In a large population with neuroimaging, CBI and WMH are independently associated with greater risks of an index fall. Increasing severities of WMH are associated incrementally with fall risk across imaging modalities.

Abstract Objective Unlocking clinical information embedded in clinical notes has been hindered to a significant degree by domain-specific and context-sensitive language. Identification of note sections and structural document elements has been shown to improve information extraction and dependent downstream clinical natural language processing (NLP) tasks and applications. This study investigates the viability of a dynamic example selection prompting method to section classification using lightweight, open-source large language models (LLMs) as a practical solution for real-world healthcare clinical NLP systems. Materials and Methods We develop a dynamic few-shot prompting approach to classifying sections where section samples are first embedded using a transformer-based model and deposited in a vector store. During inference, the embedded samples with the most similar contextual embeddings to a given input section text are retrieved from the vector store and inserted into the LLM prompt. We evaluate this technique on two datasets comprising two section schemas, including varying levels of context. We compare the performance to baseline zero-shot and randomly selected few-shot scenarios. Results The dynamic few-shot prompting experiments yielded the highest F1 scores in each of the classification tasks and datasets for all seven of the LLMs included in the evaluation, averaging a macro F1 increase of 39.3% and 21.1% in our primary section classification task over the zero-shot and static few-shot baselines, respectively. Discussion and Conclusion Our results showcase substantial performance improvements imparted by dynamically selecting examples for few-shot LLM prompting, and further improvement by including section context, demonstrating compelling potential for clinical applications.

ASCO Guidelines provide recommendations with comprehensive review and analyses of the relevant literature for each recommendation, following the guideline development process as outlined in the ASCO Guidelines Methodology Manual . ASCO Guidelines follow the ASCO Conflict of Interest Policy for Clinical Practice Guidelines . Clinical Practice Guidelines and other guidance (“Guidance”) provided by ASCO is not a comprehensive or definitive guide to treatment options. It is intended for voluntary use by clinicians and should be used in conjunction with independent professional judgment. Guidance may not be applicable to all patients, interventions, diseases or stages of diseases. Guidance is based on review and analysis of relevant literature, and is not intended as a statement of the standard of care. ASCO does not endorse third-party drugs, devices, services, or therapies and assumes no responsibility for any harm arising from or related to the use of this information. See complete disclaimer in Appendix 1 and 2 (online only) for more . PURPOSE To provide evidence-based recommendations for patients with metastatic castration-resistant prostate cancer (mCRPC). METHODS An Expert Panel including patient representation completed a systematic review of the evidence and made recommendations. RESULTS Depending upon prior treatment received, androgen receptor pathway inhibitors (ARPIs: enzalutamide, abiraterone with prednisone), poly(ADP-ribose) polymerase inhibitors (PARPi), chemotherapeutic agents (docetaxel, cabazitaxel), radiopharmaceuticals (radium 223, 177 Lu-prostate-specific membrane antigen [PSMA]-617), and sipuleucel-T have demonstrated an overall survival (OS) benefit for patients with mCRPC. For patients with BRCA1 / 2 alterations who did not receive prior ARPI, the combination of PARPi and ARPI (talazoparib + enzalutamide, olaparib and/or niraparib + abiraterone) has shown clinical benefit. For patients with BRCA1 / 2 alterations who received prior ARPI or ARPI followed by docetaxel, olaparib showed OS benefit. In select patients with microsatellite instability-high/mismatch repair-deficient, pembrolizumab showed clinical efficacy. RECOMMENDATIONS Prior systemic therapy for castration-sensitive prostate cancer will determine subsequent therapy used for mCRPC. Continue androgen-deprivation therapy for patients with mCRPC indefinitely. Early adoption of somatic genetic testing and palliative care is recommended. Patients with mCRPC and bony metastases should receive a bone-protective agent. The panel recommends the combination of ARPI with PARPi in patients with BRCA1 / 2 alterations who did not receive prior ARPI. For patients who received prior ARPI, the panel recommends docetaxel chemotherapy. The panel recommends 177 Lu-PSMA-617 or cabazitaxel chemotherapy for patients who receive prior ARPI and docetaxel chemotherapy. For patients with BRCA1 / 2 alterations who received prior ARPI, the panel recommends PARPi monotherapy. Radium 223 is recommended for patients with symptomatic bone–only disease. Evidence for optimal sequencing for mCRPC regimens is lacking. Additional information is available at www.asco.org/genitourinary-cancer-guidelines .

Abstract Objectives The surge in patient portal messages (PPMs) with increasing needs and workloads for efficient PPM triage in healthcare settings has spurred the exploration of AI-driven solutions to streamline the healthcare workflow processes, ensuring timely responses to patients to satisfy their healthcare needs. However, there has been less focus on isolating and understanding patient primary concerns in PPMs—a practice which holds the potential to yield more nuanced insights and enhances the quality of healthcare delivery and patient-centered care. Materials and Methods We propose a fusion framework to leverage pretrained language models (LMs) with different language advantages via a Convolution Neural Network for precise identification of patient primary concerns via multi-class classification. We examined 3 traditional machine learning models, 9 BERT-based language models, 6 fusion models, and 2 ensemble models. Results The outcomes of our experimentation underscore the superior performance achieved by BERT-based models in comparison to traditional machine learning models. Remarkably, our fusion model emerges as the top-performing solution, delivering a notably improved accuracy score of 77.67 ± 2.74% and an F1 score of 74.37 ± 3.70% in macro-average. Discussion This study highlights the feasibility and effectiveness of multi-class classification for patient primary concern detection and the proposed fusion framework for enhancing primary concern detection. Conclusions The use of multi-class classification enhanced by a fusion of multiple pretrained LMs not only improves the accuracy and efficiency of patient primary concern identification in PPMs but also aids in managing the rising volume of PPMs in healthcare, ensuring critical patient communications are addressed promptly and accurately.

Abstract Background Error analysis plays a crucial role in clinical concept extraction, a fundamental subtask within clinical natural language processing (NLP). The process typically involves a manual review of error types, such as contextual and linguistic factors contributing to their occurrence, and the identification of underlying causes to refine the NLP model and improve its performance. Conducting error analysis can be complex, requiring a combination of NLP expertise and domain-specific knowledge. Due to the high heterogeneity of electronic health record (EHR) settings across different institutions, challenges may arise when attempting to standardize and reproduce the error analysis process. Objectives This study aims to facilitate a collaborative effort to establish common definitions and taxonomies for capturing diverse error types, fostering community consensus on error analysis for clinical concept extraction tasks. Materials and Methods We iteratively developed and evaluated an error taxonomy based on existing literature, standards, real-world data, multisite case evaluations, and community feedback. The finalized taxonomy was released in both .dtd and .owl formats at the Open Health Natural Language Processing Consortium. The taxonomy is compatible with several different open-source annotation tools, including MAE, Brat, and MedTator. Results The resulting error taxonomy comprises 43 distinct error classes, organized into 6 error dimensions and 4 properties, including model type (symbolic and statistical machine learning), evaluation subject (model and human), evaluation level (patient, document, sentence, and concept), and annotation examples. Internal and external evaluations revealed strong variations in error types across methodological approaches, tasks, and EHR settings. Key points emerged from community feedback, including the need to enhancing clarity, generalizability, and usability of the taxonomy, along with dissemination strategies. Conclusion The proposed taxonomy can facilitate the acceleration and standardization of the error analysis process in multi-site settings, thus improving the provenance, interpretability, and portability of NLP models. Future researchers could explore the potential direction of developing automated or semi-automated methods to assist in the classification and standardization of error analysis.

Abstract Advances in genetic technology have led to the increasing use of genomic panels in precision oncology practice, with panels ranging from a couple to hundreds of genes. However, the clinical utilization and utility of oncology genomic panels, especially among vulnerable populations, is unclear. We examined the association of panel size with socioeconomic status and clinical trial matching. We retrospectively identified 9,886 eligible adult subjects in the Mayo Clinic Health System who underwent genomic testing between January 1, 2016 and June 30, 2020. Patient data were retrieved from structured and unstructured data sources of institutional collections, including cancer registries, clinical data warehouses, and clinical notes. Socioeconomic surrogates were approximated using the Area Deprivation Index (ADI) corresponding to primary residence addresses. Logistic regression was performed to analyze relationships between ADI or rural/urban status and (i) use of genomic test by panel size; (ii) clinical trial matching status. Compared with patients from the most affluent areas, patients had a lower odds of receiving a panel test (vs. a single-gene test) if from areas of higher socioeconomic deprivation [OR (95% confidence interval (CI): 0.71 (0.61–0.83), P < 0.01] or a rural area [OR (95% CI): 0.85 (0.76–0.96), P < 0.01]. Patients in areas of higher socioeconomic deprivation were less likely to be matched to clinical trials if receiving medium panel tests [(OR) (95% CI): 0.69 (0.49–0.97), P = 0.03]; however, there was no difference among patients receiving large panel tests (P > 0.05) and rural patients were almost 2x greater odds of being matched if receiving a large panel test [(OR) (95% CI): 1.76 (1.21–2.55), P < 0.01]. Significance: We identified socioeconomic and rurality disparities in the use of genomic tests and trial matching by panel size, which may have implications for equal access to targeted therapies. The lack of association between large panel tests and clinical trial matching by socioeconomic status, suggests a potential health equity impact, while removing barriers in access to large panels for rural patients may improve access to trials. However, further research is needed.

AbstractTo address the current and long‐term unmet health needs of the growing population of non‐Hodgkin lymphoma (NHL) patients, we established the Lymphoma Epidemiology of Outcomes (LEO) cohort study (NCT02736357; https://leocohort.org/). A total of 7735 newly diagnosed patients aged 18 years and older with NHL were prospectively enrolled from 7/1/2015 to 5/31/2020 at 8 academic centers in the United States. The median age at diagnosis was 62 years (range, 18–99). Participants came from 49 US states and included 538 Black/African‐Americans (AA), 822 Hispanics (regardless of race), 3386 women, 716 age <40 years, and 1513 rural residents. At study baseline, we abstracted clinical, pathology, and treatment data; banked serum/plasma (N = 5883, 76.0%) and germline DNA (N = 5465, 70.7%); constructed tissue microarrays for four major NHL subtypes (N = 1189); and collected quality of life (N = 5281, 68.3%) and epidemiologic risk factor (N = 4489, 58.0%) data. Through August 2022, there were 1492 deaths. Compared to population‐based SEER data (2015–2019), LEO participants had a similar distribution of gender, AA race, Hispanic ethnicity, and NHL subtype, while LEO was underrepresented for patients who were Asian and aged 80 years and above. Observed overall survival rates for LEO at 1 and 2 years were similar to population‐based SEER rates for indolent B‐cell (follicular and marginal zone) and T‐cell lymphomas, but were 10%–15% higher than SEER rates for aggressive B‐cell subtypes (diffuse large B‐cell and mantle cell). The LEO cohort is a robust and comprehensive national resource to address the role of clinical, tumor, host genetic, epidemiologic, and other biologic factors in NHL prognosis and survivorship.

Abstract Despite recent methodology advancements in clinical natural language processing (NLP), the adoption of clinical NLP models within the translational research community remains hindered by process heterogeneity and human factor variations. Concurrently, these factors also dramatically increase the difficulty in developing NLP models in multi-site settings, which is necessary for algorithm robustness and generalizability. Here, we reported on our experience developing an NLP solution for Coronavirus Disease 2019 (COVID-19) signs and symptom extraction in an open NLP framework from a subset of sites participating in the National COVID Cohort (N3C). We then empirically highlight the benefits of multi-site data for both symbolic and statistical methods, as well as highlight the need for federated annotation and evaluation to resolve several pitfalls encountered in the course of these efforts.