Tizhoosh, Hamid R.

Abstract Background: The increasing volume and diversity of clinical genomic reports pose a significant challenge across healthcare institutions for the interpretation and proper implementation of precision oncology research. Reports from multiple different vendors (e.g., Invitae, Ambry Genetics, Foundation Medicine) are typically PDFs and exhibit substantial heterogeneity in panel design, gene coverage, and reporting standards, which hinders efficient retrospective data mining and patient cohort identification within the electronic medical records. We sought to develop a framework to interrogate all of these reports. Methods: We extracted genomic reports from patients with breast cancer treated with neoadjuvant chemotherapy and developed MolHarmonizer, a novel, scalable framework leveraging Python and Gemini LLMs, designed to process and harmonize genomic data from disparate multi-vendor reports. Gemini LLMs are employed explicitly for robust information extraction, normalization, and structuring of key genomic features, transforming unstructured data into a unified, queryable dataset. Results: Our MolHarmonizer framework successfully processed 1,147 genomic reports from 1703 breast cancer patients (2006-2023) from 23 different companies, demonstrating robust capability to extract and standardize critical actionable biomarkers. Data sources included Invitae (n=554), Ambry Genetics (n=189), Natera (n=95), Mayo Clinic (n=88), Tempus (n=63), Guardant Health (n=47), and Foundation Medicine (n=37), with others contributing less than 20 reports. Of the samples, 827/1147 (72.1%) were germline (blood/saliva). A majority of the patients were tested using the panels due to a personal/family history (n=925). Overall, 413/1147 (36.0%) reports identified at least one mutation. For breast cancer, 75 reports showed BRCA1/2 mutations (37 BRCA1, 37 BRCA2, and one patient with both BRCA1 and 2). Other mutations identified included: PIK3CA (n=40), TP53 (n=96), PTEN (n=21), ESR1 (n=13) and AKT1/2 (n=8). Conclusion: MolHarmonizer, a powerful framework leveraging Gemini LLMs, effectively addresses genomic data heterogeneity by automating biomarker extraction and harmonization. This enables rapid cohort identification and deep retrospective analyses for clinical insights, biomarker discovery, understanding disease history, facilitating novel pattern discovery, e.g., predicting BRCA1 mutations from WSI, and accelerating research within our neoadjuvant BC cohort. Future plans include expanding to include over 20,000 breast cancer patients, developing a user-friendly chatbot, and ensuring inter-institutional adaptability for a variety of complex diseases. Citation Format: Krishna Rani Kalari, Xiaojia Tang, Thanmayee Boyapati, Tanya L. Hoskin, Sumathilatha Myla, Sumedha G. Penheiter, Richard M. Weinshilboum, Liewei Wang, Hamid R. Tizhoosh, Karthik Vikram Giridhar, Matthew P. Goetz, Judy C. Boughey. Development of an LLM framework for analysis of heterogeneous breast cancer patients genomic reports [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2026; Part 1 (Regular Abstracts); 2026 Apr 17-22; San Diego, CA. Philadelphia (PA): AACR; Cancer Res 2026;86(7 Suppl):Abstract nr 2756.

Background Breast phyllodes tumours (PT) are rare biphasic neoplasms consisting of epithelial and stromal components. They are classified into benign, borderline and malignant categories. Diagnosing and grading PTs present a challenge for pathologists, as there are multiple histological parameters and their own tiers. We aim to investigate the potential role of artificial intelligence (AI) as a diagnostic aid in determining PT grade. Materials and methods We investigated 15 PT whole slide images (WSIs), comprising 5 benign, 5 borderline and 5 malignant cases. We sought to classify and retrieve the most relevant WSIs by matching histological features at different patch sizes, using the Yottixel framework for WSI processing. Patches were extracted at a 20× magnification level. We then used the KimiaNet to extract feature vectors and transformed these into barcode representations. Barcodes of a query WSI were compared with those of others in the archive, allowing us to identify the most similar WSI and determine the PT grades from histological similarities. Results We utilized ‘majority‐n accuracy’ as a measure of correctness, that is when the majority of the top‐n search results have the correct diagnosis as the query patient. We achieved a maximum reported accuracy of 67%, with a 3000 × 3000 patch size when grading PT at majority voting with n = 4. Conclusion Despite the small sample size and absence of fine‐tuning, our study demonstrated the potential of AI‐based PT grade stratification using various patch sizes through histological matching. This serves as a preliminary proof of concept, with the prospect of refinement for potential routine clinical application.

Abstract Deep learning has advanced digital pathology and multi-omics data analysis. However, there is still a critical need to integrate these modalities to better understand complex diseases. Deep learning models, such as Prov-Gigapath, a state-of-the-art pathology foundation model, have been developed for analyzing whole-slide images (WSI). Our previously published deep-learning framework, OmicsFootPrint (OFP), transforms high-dimensional multi-omics data into intuitive genomically-arranged circular images and utilizes transfer learning to overcome the challenges of high-dimensional, low-sample-size data. In this study, we aimed to combine WSI and multi-omics data to develop a framework capable of distinguishing between invasive ductal carcinoma (IDC) and invasive lobular carcinoma (ILC), which is clinically significant as it may help explain differential responses to therapies, such as tamoxifen versus letrozole. We enhanced the performance of our OFP model by dividing each multi-omics circular image (1024x1024x3 pixels) into 16 equal-sized patches (256x256x3 pixels). Each patch was trained using a separate EfficientNetV2 architecture with pre-trained ImageNet weights. The probability outcomes from these 16 models were subsequently combined through a meta-learner model using the AutoGluon framework. Concurrently, WSI images corresponding to the same patients were independently trained, maintaining consistent train/validation/test splits. The Prov-Gigapath model, which was trained using a dataset from the Providence Health System, was employed to extract patch-level embeddings from the WSI data. These embeddings were fine-tuned using a LongNet-based masked autoencoder to predict outcomes. Finally, predictions from the OFP model and the WSI-based Prov-Gigapath model were integrated through a meta-learner, preserving the original train/validation/test split. Model performances (AUC) were evaluated using the held-out test set. The gold standard used for the analysis was histological identification of IDC versus ILC, against which the performance of omics data alone, WSI data alone, and the multi-modal integration approach were compared. We evaluated our integrated model as a proof-of-concept using The Cancer Genome Atlas (TCGA) breast cancer cohort, comprising 92 ILC and 391 IDC samples. Multi-omics data, including gene expression, microRNA, copy number variation, and RPPA, were analyzed using the OFP framework, while WSI data were processed using the Prov-Gigapath method. The results demonstrated that the multi-modal integration approach significantly outperformed individual modalities, achieving an AUC of 0.98 compared to 0.87 for OFP (original), 0.95 for OFP (16-patches), and 0.92 for WSI. Integrating WSI and multi-omics data improves predictive accuracy, significantly improving the classification of breast cancer subtypes over histology alone. Citation Format: Naresh Prodduturi, Xiaojia Tang, Hamid R. Tizhoosh, Kevin J. Thompson, Richard Weinshilboum, Karthik V. Giridhar, Judy C. Boughey, Eric W. Klee, Liewei Wang, Matthew P. Goetz, Vera Suman, Krishna R. Kalari. A multi-modal learning framework to integrate digital pathology image and multi-omics data in breast cancer [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2025; Part 1 (Regular Abstracts); 2025 Apr 25-30; Chicago, IL. Philadelphia (PA): AACR; Cancer Res 2025;85(8_Suppl_1):Abstract nr 2429.

Abstract The OmicsFootPrint framework addresses the need for advanced multi-omics data analysis methodologies by transforming data into intuitive two-dimensional circular images and facilitating the interpretation of complex diseases. Utilizing deep neural networks and incorporating the SHapley Additive exPlanations algorithm, the framework enhances model interpretability. Tested with The Cancer Genome Atlas data, OmicsFootPrint effectively classified lung and breast cancer subtypes, achieving high area under the curve (AUC) scores—0.98 ± 0.02 for lung cancer subtype differentiation and 0.83 ± 0.07 for breast cancer PAM50 subtypes, and successfully distinguished between invasive lobular and ductal carcinomas in breast cancer, showcasing its robustness. It also demonstrated notable performance in predicting drug responses in cancer cell lines, with a median AUC of 0.74, surpassing nine existing methods. Furthermore, its effectiveness persists even with reduced training sample sizes. OmicsFootPrint marks an enhancement in multi-omics research, offering a novel, efficient and interpretable approach that contributes to a deeper understanding of disease mechanisms.

Abstract Background Perianal draining tunnels in hidradenitis suppurativa (HS) and perianal fistulizing inflammatory bowel disease (IBD) present diagnostic and management dilemmas. Methods We conducted a retrospective chart review of patients with perianal disease evaluated at Mayo Clinic from January 1, 1998, through July 31, 2021. Patients' demographic and clinical data were extracted, and 28 clinical features were collected. After experimenting with several machine learning techniques, random forests were used to select the 15 most important clinical features to construct the diagnostic prediction model to distinguish perianal HS from fistulizing perianal IBD. Results A total of 263 patients were included (98 with HS, 100 with IBD, and 65 with both IBD and HS). Patients with HS had a higher mean body mass index, a higher smoking rate, and more commonly showed cutaneous manifestations of tunnels and comedones, while fistulas, abscesses, induration, anal tags, ulcers, and anal fissures were more common in patients with IBD. In addition to having lesions in the perianal area, patients with IBD often had lesions in the buttocks and perineum, while those with HS had additional lesions in the axillae and groin. Among the statistically significant features, the 15 most important were identified by random forest: fistula , tunnel , digestive symptom , knife‐cut ulcer , perineum , body mass index , age , axilla , abscess , tags , smoking , groin , genital cutaneous edema , erythema , and bilateral/unilateral . Conclusions The results of this study may help differentiate perianal lesions, especially perineal HS and fistulizing perineal IBD, and provide promise for a better therapeutic outcome.

Abstract Binary encoding of features has proven to be a versatile tool for optimizing data processing and memory efficiency in various machine learning applications. This includes deep barcoding of images for retrieval of similar cases among millions of indexed images. Despite the recent advancement in barcode generation methods, converting hyperdimensional feature vectors (e.g., deep features) to compact and discriminative binary barcodes is still an urgent necessity and widely an unresolved problem. MinMax barcoding of features is one of the efficient binarization methods, where the accuracy of the generated barcodes is inherently dependent on the feature order due to its computation process. In this paper, we propose a combinatorial evolutionary framework for optimizing barcodes. The primary goal is to determine an optimal permutation of extracted deep features, leading to more accurate barcodes. The performance of the proposed model has been assessed in downstream image retrieval tasks using a variety of datasets. This evaluation includes medical images from The Cancer Genome Atlas (TCGA), a comprehensive publicly available dataset, as well as images from a COVID-19 dataset. In addition, the model's performance was tested on diverse non-medical image collections, such as CIFAR-10 and Fashion-MNIST. Our findings demonstrate that optimizing barcodes significantly enhances retrieval accuracy across a wide range of applications, highlighting the applicability and potential of barcodes in various domains.

Abstract We present SEQuence Weighted Alignment for Sorting and Harmonization (Seqwash), an algorithm designed to process sequencing profiles utilizing large language models. Seqwash harmonizes immune cell sequences into a unified representation, empowering LLMs to embed meaningful patterns while eliminating irrelevant information. Evaluations using immune cell sequencing data showcase Seqwash’s efficacy in standardizing profiles, leading to improved feature quality and enhanced performance in both supervised and unsupervised downstream tasks for sequencing data.