Osborne, Collin

Abstract Scientific foundation models learn high-dimensional representations from diverse data modalities, yet what they encode and how to extract that knowledge remain open questions. Here we show that probing the internal representations of Evo 2, a 7-billion-parameter genomic foundation model, enables accurate and interpretable genetic variant effect prediction. We introduce a covariance-based probe that captures second-order structure from Evo 2 sequence embeddings to predict variant pathogenicity across variant types and functional consequences, matching or exceeding specialized predictors within their domains. To ground these predictions in known biological mechanisms, we train a complementary panel of probes on existing annotations to detect which genomic properties are disrupted by a variant. This categorized evidence is then integrated with each variant’s genomic context through a language model to generate variant-specific mechanistic hypotheses. Our pathogenicity predictions correlate with experimental measures of variant function, clinical penetrance, and biobank disease associations while the mechanistic hypotheses are consistent with expert reviews, known mechanism classes, and downstream molecular readouts. We release pathogenicity scores, disruption profiles, and contextualized interpretations for 4.2 million variants from the ClinVar database as an open resource through the Evo Variant Effect Explorer (EVEE). More broadly, this structured probing approach offers a general framework for interrogating foundation models across scientific disciplines and grounding their outputs in existing domain concepts.

Abstract The advent of high-throughput sequencing has led to an exponential increase in genomic data, highlighting the need for efficient and accurate models to analyze and interpret this information. In this study, we introduce a novel, exomic foundational model that leverages a combination of the human reference genome and multispecies data to improve variant detection and interpretation. Our model utilizes a short-range transformer architecture and is trained on a large dataset of human exomic sequences derived from the Tapestry study. Through a series of ablation studies and scaling experiments, we demonstrate the effectiveness of our model in predicting next token accuracy and identifying clinically pathogenic variants. We also show that our model outperforms existing models in a range of downstream tasks, including variant effect prediction and disease state identification. In fact, our largest sequence transformer for range adaptive nucleotide decoding variant (1B parameters) surpassed previous benchmarks, demonstrating a mean accuracy of 0.880 [an 8.2% improvement over the original nucleotide transformer (NT) and a 7% improvement over NT-v2]. Furthermore, we construct a unique exomic ClinVar dataset to evaluate the model’s performance on pathogenicity and disease states. Our results highlight the potential of this model to improve our understanding of the human exome and its role in disease. The model and its applications have significant implications for genomics-based diagnosis and personalized medicine, including tailored therapeutic development.

Abstract The advent of high-throughput sequencing has led to an exponential increase in genomic data, highlighting the need for efficient and accurate models to analyze and interpret this information. Here, we introduce a novel, exomic foundational model that leverages a combination of human reference genome and multispecies data to improve variant detection and interpretation. Our model utilizes a short- range transformer architecture and is trained on a large dataset of human exomic sequences derived from the Tapestry study. Through a series of ablation studies and scaling experiments, we demonstrate the effectiveness of our model in pre- dicting next token accuracy and identifying clinically pathogenic variants. We also show that our model outperforms existing models in a range of downstream tasks, including variant effect prediction and disease state identification. In fact, our largest STRAND variant (1B parameters) surpassed previous benchmarks, demonstrating a mean accuarcy of 0.880 (8.2% improvement over the original NT and 7% improvement over NT-v2). Furthermore, we construct a unique exomic ClinVar dataset to evaluate the model’s performance on pathogenicity and disease states. Our results highlight the potential of this model to improve our understanding of the human exome and its role in disease. The model and its applications have significant implications for genomic based diagnosis and personalized medicine including tailored therapeutic development.

The differentiation of B cells into antibody secreting plasma cells (PCs) is governed by a strict regulatory network that results in expression of specific transcriptomes along the activation continuum. In vitro models yielding significant numbers of PCs phenotypically identical to the in vivo state enable investigation of pathways, metabolomes, and non-coding (ncRNAs) not previously identified. The objective of our study was to characterize ncRNA expression during human B cell activation and differentiation. To achieve this, we used an in vitro system and performed RNA-seq on resting and activated B cells and PCs. Characterization of coding gene transcripts, including immunoglobulin (Ig), validated our system and also demonstrated that memory B cells preferentially differentiated into PCs. Importantly, we identified more than 980 ncRNA transcripts that are differentially expressed across the stages of activation and differentiation, some of which are known to target transcription, proliferation, cytoskeletal, autophagy and proteasome pathways. Interestingly, ncRNAs located within Ig loci may be targeting both Ig and non-Ig-related transcripts. ncRNAs associated with B cell malignancies were also identified. Taken together, this system provides a platform to study the role of specific ncRNAs in B cell differentiation and altered expression of those ncRNAs involved in B cell malignancies.