Protein Language Models: ESM2, ProtTrans, and What They Can Predict About Protein Function

Protein language models such as ESM2 and ProtTrans apply the same architecture behind large language models to amino acid sequences instead of words. Rather than folding a protein into a three-dimensional shape, these models extract evolutionary and functional signal directly from sequence alone, predicting properties that inform everything from enzyme engineering to variant interpretation.

What protein language models are

Protein language models apply the transformer architecture used in natural language processing directly to amino acid sequences, treating each residue the way a text model treats a word. Instead of learning grammar and meaning from sentences, the model learns evolutionary and structural constraints from millions of curated protein sequences, building a numerical representation, or embedding that captures biologically meaningful relationships between residues and across whole proteins.

The training data is the key distinction from a general-purpose chatbot. Protein language models are trained on curated sequence repositories such as UniRef rather than on internet text, and the self-supervised objective typically involves masking part of a sequence and asking the model to predict the missing residues from their context. This masked-language objective forces the model to internalize patterns of amino acid co-occurrence that reflect real evolutionary and structural constraints, even though no structural label is ever provided during training.

The resulting embeddings are dense numerical vectors that encode information a researcher can feed into downstream predictive tasks. A biochemist does not need to interpret the embedding directly; instead, a simple classifier or regression model, built using the kind of machine learning methods guide researchers now rely on across biology, can predict a specific property, such as subcellular localization or catalytic activity, often capturing patterns that hand-crafted sequence features alone would miss.

Embeddings come in two practical forms that serve different research needs. Per-residue embeddings preserve a separate vector for every amino acid position, which is useful for tasks that depend on local context, such as identifying a catalytic residue or a binding site. Per-protein embeddings collapse that information into a single vector, typically through an averaging or pooling operation, which is better suited to whole-protein comparisons such as function classification or homology search across large sequence databases.

How ESM2 and ProtTrans learn from sequence data

ESM2, developed by the protein team at Meta AI, is trained purely on sequence data using a masked-language objective similar to BERT-style natural language models, with no structural information supplied during training. As transformer protein language models were trained with up to 15 billion parameters on experimental and high-quality predicted structures, researchers observed that information about atomic-level structure emerged in the model as it was scaled up, a finding reported in a scaled protein language model study in Science.

ProtTrans, built by the Rostlab research group at the Technical University of Munich, takes a related but architecturally distinct approach. The project trained several auto-regressive and auto-encoder model families, including variants based on BERT, T5, and XLNet, on sequence data from UniRef and the Big Fantastic Database, reaching a combined training corpus of up to 393 billion amino acids in a self-supervised learning of proteins effort designed to test how far pure sequence pretraining could go.

Both models share the same underlying logic despite their architectural differences. Neither ESM2 nor ProtTrans ever sees a labeled structure or an annotated function during pretraining; the biological signal these models capture comes entirely from patterns of sequence conservation and covariation across evolutionarily related proteins. That sequence-only design is also what makes their predictions portable across species and protein families that lack solved structures.

Both model families are also released at multiple parameter scales rather than as a single fixed checkpoint. ESM2 spans versions from roughly 8 million parameters up to the 15-billion-parameter model described in its scaling analysis, and ProtTrans similarly offers smaller and larger variants across its BERT, T5, and XLNet-based architectures. This range matters practically, since a smaller checkpoint can run on a single graphics processing unit for rapid screening, while the largest checkpoints generally provide higher accuracy at the cost of substantially more compute.

What protein language models predict about function

Protein language models predict a range of functional properties directly from sequence, including secondary structure, subcellular localization, membrane vs soluble character, and enzyme function, once embeddings from ESM2 or ProtTrans are paired with a lightweight supervised model trained on labeled examples. This sequence-only workflow avoids the need for a solved structure or a family-specific alignment before a prediction can be made.

One of the more distinctive capabilities is zero-shot variant effect prediction, in which a protein language model estimates how a mutation will affect protein function without any task-specific training on experimental fitness data. Researchers behind a zero-shot mutation effect prediction study demonstrated that protein language models capture the functional effects of sequence variation using only zero-shot inference, performing competitively against methods trained on family-specific alignments.

Comparing performance across methods requires a shared evaluation framework, and the field has largely converged on Gene Ontology-based benchmarking for function prediction through the recurring CAFA challenge, which has continued through multiple rounds since it began. The first large-scale function prediction evaluation, published in 2013, found that the computational methods tested at the time substantially outperformed widely used first-generation, similarity-based approaches, though the same evaluation noted considerable room for improvement, particularly for predicting broader biological process terms rather than narrower molecular functions. Claims about any single protein language model being definitively superior to competing methods should be read with this caveat in mind, since performance varies by task, protein family, and evaluation metric, and rankings have continued to shift across the CAFA rounds that have followed.

Protein engineering is one of the application areas where these predictions have the most direct practical value. A protein language model can score thousands of candidate mutations computationally before a single one is synthesized in the laboratory, narrowing a directed evolution campaign to the variants most likely to improve stability, activity, or binding affinity. This computational triage step does not replace experimental validation, but it can meaningfully reduce the number of variants that need to be tested at the bench.

How protein language models compare to AlphaFold

AlphaFold answers a different question than a protein language model does. Rather than predicting function or fitness from sequence, AlphaFold predicts the three-dimensional coordinates of a protein's atoms, and its developers demonstrated accuracy competitive with experimental structures in a majority of cases through a near-experimental accuracy structure prediction approach validated in the Critical Assessment of protein Structure Prediction. AlphaFold relies on multiple sequence alignments (MSAs) and, in most configurations, benefits from having evolutionarily related sequences available, whereas protein language models can generate a prediction from a single sequence with no alignment step required.

Figure 1: A two-panel comparison showing how a protein sequence flows into a protein language model to predict function versus into a structure predictor to generate a three-dimensional model. Credit: AI-generated image created using Google Gemini (2026).

The practical distinction matters for how a researcher chooses a tool. A question about what a protein does, or how a mutation might change its behavior, is best answered with a protein language model. A question about the physical shape a protein adopts, needed for docking studies or mechanistic interpretation, calls for AlphaFold or a comparable structure predictor. The two approaches are complementary rather than competing, and many current research pipelines apply both in sequence, using a protein language model for functional triage before committing computational resources to full structure prediction.

Computational cost is a practical factor behind that sequencing decision. Generating a protein language model embedding for a single sequence typically takes seconds to a few minutes depending on checkpoint size, while a full structure prediction, particularly for a long protein without strong sequence homologs, can require substantially more compute time and, in most configurations, an MSA search step that a protein language model does not need. Running the faster, alignment-free prediction first is a practical way to prioritize which candidates justify the added cost of full structure prediction.

A third category extends this landscape from prediction to creation. Generative protein design tools apply diffusion-based and other generative architectures to produce entirely new sequences and structures to a functional specification, rather than analyzing an existing protein. One such generative de novo protein design framework demonstrated the ability to generate diverse functional proteins from simple molecular specifications, extending the sequence-to-function and sequence-to-structure logic of language models and AlphaFold into de novo engineering.

Using protein language models: Practical tools and workflow considerations

Working with a protein language model in practice typically means generating embeddings from a pretrained checkpoint rather than training a model from scratch, a workflow that fits within the broader AI and data science landscape now shaping computational biology research. Both ESM2 and ProtTrans are distributed as open research tools with pretrained weights available at multiple parameter scales, letting a research group choose a smaller, faster model for exploratory work or a larger model when prediction accuracy matters more than runtime.

Synthetic Biology: Engineering Life

This infographic provides a clear overview of the workflow, technologies, and applications shaping synthetic biology.
View Infographic

Advertisement

Computational requirements scale with model size, and the largest available checkpoints require substantial graphics processing unit memory to run efficiently, which is a practical constraint smaller labs should weigh against the accuracy gains a larger model offers. Sequence-only input is also a genuine operational advantage over alignment-dependent methods, since a protein language model can produce an embedding immediately without the additional runtime and database dependencies that MSA construction requires.

Reproducibility considerations common to other computational biology tools matter here as well. Recording the exact checkpoint version, embedding extraction layer, and any pooling method used is essential, since predictions from an earlier or later model version, or from a different embedding layer within the same model, are not guaranteed to match. Documenting these choices alongside any downstream classifier's training data and hyperparameters gives a research group a reproducible record it can revisit if a prediction later needs rechecking against new experimental data.

A short, practical framework helps a research team choose the right tool for a specific protein science question:

1. Identify the question type. Function, fitness, and mutation effect questions point toward a protein language model; shape and docking questions point toward a structure predictor.
2. Check what training data is required. Zero-shot protein language model predictions need no labeled data; supervised downstream tasks need a labeled training set for the specific property of interest.
3. Match model size to available compute. Smaller checkpoints are appropriate for exploratory screening; larger checkpoints are appropriate when a single high-confidence prediction is worth the added runtime.
4. Validate against experimental data wherever possible. A predicted function, fitness effect, or structure is a hypothesis that still benefits from laboratory confirmation before it informs a major research decision.
5. Combine tools where the question demands it. Many workflows apply a protein language model for initial triage before committing structure-prediction or generative-design resources to the most promising candidates.

Table 1: A qualitative comparison of protein language models, structure predictors, and generative protein design tools by primary research question and output.

Protein language models like ESM2 give research a sequence-first tool

Protein language models such as ESM2 and ProtTrans give researchers a way to extract functional and evolutionary signal directly from sequence, without requiring a solved structure or a family-specific alignment. That sequence-first approach makes them a practical complement to structure predictors like AlphaFold rather than a replacement for them, since the two model types answer fundamentally different biological questions.

As generative design tools extend this landscape further, understanding what each category of model can and cannot predict is becoming a core literacy for anyone working at the intersection of computational and structural biology. Protein science increasingly draws on the broader momentum behind large language models in biology, and researchers who understand where sequence-based prediction, structure prediction, and generative design each fit are better positioned to choose the right tool for a given question rather than defaulting to whichever method is most discussed.

This content includes text that has been created with the assistance of generative AI and has undergone editorial review before publishing. Technology Networks' AI policy can be found here.