Pfaff, Emily

Abstract Despite recent methodology advancements in clinical natural language processing (NLP), the adoption of clinical NLP models within the translational research community remains hindered by process heterogeneity and human factor variations. Concurrently, these factors also dramatically increase the difficulty in developing NLP models in multi-site settings, which is necessary for algorithm robustness and generalizability. Here, we reported on our experience developing an NLP solution for Coronavirus Disease 2019 (COVID-19) signs and symptom extraction in an open NLP framework from a subset of sites participating in the National COVID Cohort (N3C). We then empirically highlight the benefits of multi-site data for both symbolic and statistical methods, as well as highlight the need for federated annotation and evaluation to resolve several pitfalls encountered in the course of these efforts.

e13552 Background: Real-world data from Electronic Health Records (EHR) have been widely used for patient identification to build study cohorts for clinical research. Traditionally, diagnosis codes in the EHR, such International Classification of Diseases (ICD), are used to identify the target patients. However, the accuracy of this approach is dependent on the accuracy of ICD coding, with potential errors especially for tumor types that are frequent locations for metastases (which may contribute to mis-coding). In this study, we attempted to develop a Machine Learning (ML) based approach on EHR data to improve the accuracy of identification of patients with lung cancer. Methods: We used survey respondents in the Enhanced, EHR-facilitated Cancer Symptom Control (E2C2, NCT03892967) cluster-randomized trial at Mayo Clinic as our initial pan-cancer cohort. E2C2 includes adults receiving Medical Hematology/Oncology care for a solid or liquid tumor at Mayo Clinic. We collected cancer diagnoses from the individually abstracted Mayo Clinic Cancer Registry to annotate the cancer type for the patients. Lung cancer related ICD-9 (162.X) and ICD-10 (C34.X) codes were used to build a search query on the Mayo’s EHR to find target patients from the E2C2 cohort, and to investigate the ICD-based lung cancer patient identification performance. Diagnosis, radiation oncology treatment (CPT 77261 - 77799), and antineoplastic drug administration data were collected from EHR as variables. Logistic Regression (LR), Support vector machine (SVM), Random Forest (RF), and eXtreme Gradient Boosting (XGB) were selected to build models for lung cancer patient identification. 10-fold cross-validation was implemented to assess the models. Precision, Recall, F1 Score, and area under the curve (AUC) were selected to measure the performance. Results: We collected 13,893 patients with a specific cancer diagnosis, and 1,394 were identified as having lung cancer. The identification performance across different methods is shown. The ICD-based method only had 0.65 precision. It means we collected a lot of false positive cases (other cancer but no lung cancer patient), as we mentioned in the background. SVM gets the best precision results, but its recall and F1 score are not good enough. XGB shows the best F1 Score and AUC, which also means this method achieved the best and most balanced performance. Conclusions: In this study, we found that XGB-based methods achieved the best identification performance for lung cancer. In the future, we will investigate if this is also true for the identification of other cancer types. [Table: see text]