Foundation Models for Genomics: What DNA-BERT, Nucleotide Transformer, and Evo Mean for Research

Foundation models, large neural networks pre-trained on massive unlabeled datasets and adapted for specific tasks, have reshaped natural language processing (NLP). The same approach is now being applied to genomic sequences, producing genomic foundation models trained on DNA at scale to learn representations that transfer across biological problems. Understanding what these models actually do, and where they fall short, is increasingly relevant across genomics and computational biology.

Applying the foundation model concept to genomic sequences

A foundation model is a large machine learning model pre-trained on broad, unlabeled data and fine-tuned for specific downstream tasks. In NLP, this paradigm is exemplified by BERT and GPT, which learn generalizable linguistic representations before being adapted to tasks such as sentiment analysis or question answering. Genomic foundation models apply the same logic to DNA: instead of words, the input is nucleotide sequence; instead of semantic meaning, the model learns biological representations of functional elements, conservation patterns, and regulatory grammar.

Figure 1: A five-step flowchart illustrating the pre-training and fine-tuning workflow of a genomic foundation model. Credit: AI-generated image created using Google Gemini (2026).

The transfer learning framework offers a meaningful practical advantage in genomics. Labeled datasets for regulatory element annotation require extensive experimental work, from chromatin immunoprecipitation assays to massively parallel reporter experiments. A model pre-trained on the vast quantity of available unlabeled genomic sequence can capture sequence-level patterns that are difficult to encode in hand-crafted features, then be fine-tuned with modest labeled data. This is particularly useful for low-data biological prediction tasks, and it is one reason researchers building AI-driven genomics workflows are adopting these architectures alongside traditional bioinformatics methods.

The defining characteristic of a genomic foundation model is that its pre-training is sequence-driven and unsupervised, without biological annotations. The model learns to predict masked or subsequent nucleotides from context, much as a language model learns to predict masked words. The result is a set of learned representations encoding biological signal implicitly, enabling transfer to tasks such as predicting promoter activity, identifying splice sites, or scoring the effect of single-nucleotide variants (SNVs) on gene regulation.

How genomic foundation models are trained

The training process begins with a large corpus of DNA sequences gathered from reference genomes, population sequencing datasets, or whole-genome assemblies spanning multiple species. The model then learns to reconstruct masked or held-out nucleotides, a task known as masked language modeling (MLM), which forces it to encode the broader sequence context needed to make accurate predictions. Generative models such as Evo instead train autoregressively, predicting each nucleotide given all preceding ones.

Tokenization, the process of dividing raw sequence into the basic units a model processes, is a defining engineering choice. Early models such as DNA-BERT used k-mer tokenization, dividing the sequence into fixed-length overlapping substrings of length k. While conceptually simple, overlapping k-mers introduce data leakage between adjacent tokens and can artificially inflate benchmark performance. DNABERT-2 replaced k-mer tokenization with byte pair encoding (BPE), a compression-based approach borrowed from NLP that builds a vocabulary of variable-length sequence fragments based on statistical co-occurrence in the training corpus. BPE avoids overlapping-token artifacts and yields more efficient representations.

The choice of training corpus shapes what a model can and cannot represent. Models trained exclusively on the human reference genome cannot learn interspecies conservation, limiting their utility for comparative genomics. Multi-species training, as in the Nucleotide Transformer, expands the learned representation by exposing the model to evolutionary diversity across thousands of genomes. Evo takes a fundamentally different approach, training on prokaryotic and phage genomes at whole-genome scale, which enables modeling of long-range genome organization but restricts direct applicability to eukaryotic biology without additional fine-tuning.

Key models: DNA-BERT, Nucleotide Transformer, and Evo

The genomic foundation model landscape spans several architectures that differ substantially in scale, context length, and training approach. The table below summarizes the principal properties of the most widely used models.

Table 1: Selected properties of five genomic foundation models spanning the current landscape of architecture, context length, and training corpus.

The original DNA-BERT architecture, introduced in 2021, was among the first applications of the BERT architecture to genomic sequences. Trained on the human reference genome using 6-mer tokenization, it demonstrated strong performance on regulatory element classification tasks including splice site prediction, promoter identification, and transcription factor (TF) binding site prediction. DNA-BERT established that the transfer learning paradigm from NLP could be meaningfully applied to DNA, and it became the benchmark baseline against which subsequent models are measured. Its successor, DNABERT-2, replaced k-mer tokenization with BPE and introduced multi-species training, achieving comparable performance with substantially fewer parameters and reduced pretraining compute.

The Nucleotide Transformer, published in Nature Methods, scaled the foundation model approach substantially. The model family spans 50 million to 2.5 billion parameters and integrates information from 3,202 human genomes alongside 850 genomes from diverse phyla, including both model and non-model organisms. In zero-shot evaluation, Nucleotide Transformer representations match or outperform specialized methods on 11 of 18 prediction tasks; after fine-tuning, that figure rises to 15 of 18. The availability of models across a wide range of parameter budgets makes the Nucleotide Transformer a practical starting point for researchers with varying computational resources.

Evo takes a divergent approach, applying the StripedHyena architecture, a hybrid of attention and data-controlled convolutional operators, to whole-genome sequence modeling. With 7 billion parameters and a context length of 131 kilobases (kb), Evo processes sequences at single-nucleotide, byte-level resolution with near-linear compute scaling relative to sequence length. It was trained on a dataset comprising 300 billion nucleotides from prokaryotic and phage genomes, enabling scaling laws on DNA analogous to those established in NLP and computer vision. HyenaDNA, a precursor model built on the Hyena convolutional architecture, demonstrated that abandoning the quadratic attention mechanism enables million-token single-nucleotide modeling, reaching state-of-the-art performance on 12 of 18 Nucleotide Transformer benchmark datasets with far fewer parameters.

What genomic foundation models can predict

Genomic foundation models address a range of prediction tasks across regulatory genomics, variant biology, and genome annotation. The most established applications involve regulatory element classification: predicting whether a given sequence functions as a promoter, enhancer, splice site, or TF binding site. These tasks are well-suited to encoder models such as DNA-BERT and the Nucleotide Transformer, which learn bidirectional sequence representations from the full context of an input window.

Variant effect prediction is a closely related application. By comparing model representations or log-likelihood scores between a reference and an alternative allele, genomic foundation models can score the likely functional impact of SNVs without task-specific training data. This zero-shot scoring approach is particularly useful for interpreting variants in noncoding regulatory regions, where experimental functional data remain sparse. In the context of candidate regulatory variants from genome-wide association studies, model-derived scores offer a scalable route to prioritization before experimental follow-up.

Evo extends the predictive repertoire substantially. The model demonstrates zero-shot function prediction competitive with domain-specific language models across DNA, RNA, and protein modalities, reflecting what becomes accessible when context length extends to tens of kilobases: systems-level sequence grammar that shorter-context models cannot capture. Notably, Evo has generated functional CRISPR-Cas and transposon systems from sequence alone, representing the first demonstrations of protein-RNA and protein-DNA co-design using a language model. The model also learns how small mutations affect whole-organism fitness, enabling mutational scanning across the full protein-coding space of a genome.

How genomic foundation models perform on standard benchmarks

Genomic foundation model performance is typically evaluated on standardized task suites. The 18-task Nucleotide Transformer benchmark covers regulatory elements including histone modifications, enhancer and promoter prediction, and splice site prediction. The Genome Understanding Evaluation (GUE), introduced alongside DNABERT-2, spans 36 datasets across nine task types applied to multiple species including human, mouse, and yeast. Independent benchmarking across five models including DNABERT-2, Nucleotide Transformer, HyenaDNA, Caduceus, and GROVER shows that model performance varies substantially across tasks and datasets, with no single model dominating all benchmarks.

Advancing AAV Characterization Through Mass Photometry

This infographic shows how single-particle mass photometry delivers AAV purity analytics in less than five minutes while supporting development and GMP manufacturing workflows.
View Infographic

Advertisement

HyenaDNA reaches state-of-the-art results on 12 of 18 Nucleotide Transformer benchmark datasets using a model with far fewer parameters, suggesting that architectural efficiency and single-nucleotide resolution can sometimes outweigh raw model scale. Comparing results across published papers is complicated by varying fine-tuning protocols, embedding strategies, and dataset splits. A 2024 benchmarking study identified that mean token embedding consistently improves sequence classification performance over other pooling strategies, a methodological factor that confounds cross-paper comparisons independently of model quality.

No universal evaluation framework yet exists that would allow a researcher to definitively select the best model for a given task without empirical testing. Benchmarks are typically designed around short input windows of a few hundred to a few thousand nucleotides, which excludes assessment of the long-range dependencies that distinguish convolutional and hybrid architectures from BERT-style transformers. Performance reported in model papers should therefore be treated as a lower bound on what fine-tuning can achieve, not as a guide to task-specific selection.

Limitations and practical considerations for genomic foundation models

Despite strong benchmark performance, genomic foundation models carry important limitations. Most leading models were trained on human reference genomes or prokaryotic datasets. Models optimized for human genomics, such as DNA-BERT and the Nucleotide Transformer, may generalize poorly to organisms with substantially different genome organization, repeat content, or regulatory architecture. Evo, trained on prokaryotic and phage genomes, has limited direct applicability to eukaryotic gene regulation.

Context length creates a ceiling on addressable biology. Enhancers, topologically associating domains (TADs), and other regulatory elements can interact across hundreds of kilobases, a scale that 512-token BERT-style models cannot represent. Long-context architectures address this, but their training on non-eukaryotic data limits immediate application to human disease biology. All current genomic foundation models are also largely black-box predictors: while attention-based models allow visualization of token importance, mechanistic interpretability remains limited, and it is unclear whether model representations correspond to biological sequence grammar or statistical artifacts of training data.

Selecting and deploying a genomic foundation model involves several practical steps:

1. Define the prediction task precisely, specifying input sequence length, organism, and output label type before model selection.
2. Identify whether a relevant pre-trained model exists for your organism, sequence window, and primary task category.
3. Evaluate multiple candidate models on a held-out biological validation set rather than relying on published benchmark scores derived from different datasets.
4. Apply parameter-efficient fine-tuning methods such as low-rank adaptation (LoRA) where labeled data are limited, to reduce training compute while retaining pre-trained representations.
5. Audit predictions for systematic biases introduced by training data composition, particularly at repetitive or poorly assembled genomic regions.

Genomic foundation models and the future of sequence-based research

The trajectory of genomic foundation models closely mirrors that of protein language models a few years earlier: rapid increases in scale, architectural innovation, and expanding task coverage. Researchers navigating this landscape can draw on the broader map of AI in life science research to frame where sequence models fit within the wider computational toolkit. The field is moving toward models that handle longer contexts, integrate multiple sequence modalities, and generalize across the tree of life rather than being constrained to human or microbial genomes.

For researchers, the practical implication is that genomic foundation models are most useful as starting points rather than drop-in solutions. A model pre-trained on the right corpus, fine-tuned with appropriately curated data, and evaluated against biologically meaningful criteria will outperform any default benchmark result. The substantive advances are real: faster regulatory annotation, zero-shot variant scoring, and the first ML-driven generation of functional genetic systems. Calibrating expectations to what is currently achievable, rather than what might eventually be possible, determines how productively these tools are deployed in working research contexts.

This content includes text that has been created with the assistance of generative AI and has undergone editorial review before publishing. Technology Networks' AI policy can be found here.