Transformers vs. State Space Models for Genomic Representation Learning: Orthology Detection, Latent Space Alignment, and Trade-offs of Long-Context Modeling

This thesis investigates the application of biological foundation models to the task of ortholog identification. Ortholog datasets were constructed from National Center for Biotechnology Information and OMA by selecting eukaryotic species with uniform evolutionary distance from humans and organizing genes into ortholog families. An evolutionary-aware contrastive learning loss was developed to incorporate phylogenetic information into representation learning. Embeddings extracted from Nucleotide Transformer v2 and Evo 2 were used to train a lightweight contrastive projection head for orthology prediction. Different sequence configurations were evaluated, including only coding regions, regulatory regions, and full genomic sequences. The work also includes representation alignment analyses through contrastive learning and preliminary scaling law evaluations for Evo 2.