Liu, Hongfang

Abstract Background Integrating advanced artificial intelligence (AI) into clinical decision-support often requires the sharing of sensitive patient data with external services, raising privacy concerns. Homomorphic encryption (HE) allows computing directly on encrypted data, without revealing the underlying patient information. Objectives To develop a large language model (LLM)-assisted diagnosis framework while preserving patient privacy in the clinical text analysis, by leveraging HE and using rare disease (RD) diagnosis as a representative application. To demonstrate HE does not hinder the system performance. Materials and Methods Texts from patient histories and a RD knowledge base were embedded by LLMs into vectors, then encrypted using HE to obscure private information while retaining the semantic nuances. Diagnostic recommendations were generated by computing and ranking the similarities between the patient history and RD vectors in the encrypted space. The system was evaluated using 50 synthetic case reports (5 RDs, each with 10 reports). Results Applying HE did protect private information from reverse-embedding attacks. HE imposed little disruption to the diagnostic accuracy, with normalized discounted cumulative gains (nDCG) of 0.6108 ± 0.3412 (encrypted) versus 0.6083 ± 0.3415 (unencrypted). The accuracy and computational performance were tunable and consistent, as demonstrated across five different LLMs. Discussion Our privacy-preserving framework opens tremendous opportunities toward hosting and serving powerful AI solutions across institution boundaries, which would remove the need for local deidentification and incentivize users to access secure external decision-support services. Conclusions Integrating HE with LLM retrieval can promote the dissemination of nonredundant, high-capacity AI services by preserving both privacy and accuracy.

Abstract Current discussion surrounding the clinical capabilities of generative language models (GLMs) predominantly center around multiple-choice question-answer (MCQA) benchmarks derived from clinical licensing examinations. While accepted for human examinees, characteristics unique to GLMs bring into question the validity of such benchmarks. Here, we validate four benchmarks using eight GLMs, ablating for parameter size and reasoning capabilities, validating via prompt permutation three key assumptions that underpin the generalizability of MCQA-based assessments: that knowledge is applied, not memorized, that semantic consistency will lead to consistent answers, and that situations with no answers can be recognized. While large models are more resilient to our perturbations compared to small models, we globally invalidate these assumptions, with implications for reasoning models. Additionally, despite retaining the knowledge, small models are prone to answer memorization. All models exhibit significant failure in null-answer scenarios. We then suggest several adaptations for more robust benchmark designs more reflective of real-world conditions.

Abstract Advances in genetic technology have led to the increasing use of genomic panels in precision oncology practice, with panels ranging from a couple to hundreds of genes. However, the clinical utilization and utility of oncology genomic panels, especially among vulnerable populations, is unclear. We examined the association of panel size with socioeconomic status and clinical trial matching. We retrospectively identified 9,886 eligible adult subjects in the Mayo Clinic Health System who underwent genomic testing between January 1, 2016 and June 30, 2020. Patient data were retrieved from structured and unstructured data sources of institutional collections, including cancer registries, clinical data warehouses, and clinical notes. Socioeconomic surrogates were approximated using the Area Deprivation Index (ADI) corresponding to primary residence addresses. Logistic regression was performed to analyze relationships between ADI or rural/urban status and (i) use of genomic test by panel size; (ii) clinical trial matching status. Compared with patients from the most affluent areas, patients had a lower odds of receiving a panel test (vs. a single-gene test) if from areas of higher socioeconomic deprivation [OR (95% confidence interval (CI): 0.71 (0.61–0.83), P < 0.01] or a rural area [OR (95% CI): 0.85 (0.76–0.96), P < 0.01]. Patients in areas of higher socioeconomic deprivation were less likely to be matched to clinical trials if receiving medium panel tests [(OR) (95% CI): 0.69 (0.49–0.97), P = 0.03]; however, there was no difference among patients receiving large panel tests (P > 0.05) and rural patients were almost 2x greater odds of being matched if receiving a large panel test [(OR) (95% CI): 1.76 (1.21–2.55), P < 0.01]. Significance: We identified socioeconomic and rurality disparities in the use of genomic tests and trial matching by panel size, which may have implications for equal access to targeted therapies. The lack of association between large panel tests and clinical trial matching by socioeconomic status, suggests a potential health equity impact, while removing barriers in access to large panels for rural patients may improve access to trials. However, further research is needed.

Abstract Rationale Difficulty of asthma ascertainment and its associated methodologic heterogeneity have created significant barriers to asthma care and research. Objectives We evaluated the validity of an existing natural language processing (NLP) algorithm for asthma criteria to enable an automated chart review using electronic medical records (EMRs). Methods The study was designed as a retrospective birth cohort study using a random sample of 500 subjects from the 1997–2007 Mayo Birth Cohort who were born at Mayo Clinic and enrolled in primary pediatric care at Mayo Clinic Rochester. Performance of NLP-based asthma ascertainment using predetermined asthma criteria was assessed by determining both criterion validity (chart review of EMRs by abstractor as a gold standard) and construct validity (association with known risk factors for asthma, such as allergic rhinitis). Measurements and Main Results After excluding three subjects whose respiratory symptoms could be attributed to other conditions (e.g., tracheomalacia), among the remaining eligible 497 subjects, 51% were male, 77% white persons, and the median age at last follow-up date was 11.5 years. The asthma prevalence was 31% in the study cohort. Sensitivity, specificity, positive predictive value, and negative predictive value for NLP algorithm in predicting asthma status were 97%, 95%, 90%, and 98%, respectively. The risk factors for asthma (e.g., allergic rhinitis) that were identified either by NLP or the abstractor were the same. Conclusions Asthma ascertainment through NLP should be considered in the era of EMRs because it can enable large-scale clinical studies in a more time-efficient manner and improve the recognition and care of childhood asthma in practice.