Protein Set Transformer (PST) is a protein-based genome language model that represents genomes as sets of proteins to improve genome and protein representations across diverse viral datasets
0:00Welcome to Base by Base, the papercast that brings genomics to you wherever you are. Thanks for listening, and don't forget to follow and rate us in your podcast app. Today, we are embarking on a deep dive into the world of viruses.
0:12Those microscopic masters of mutation. And, I mean, just think about this for a second. Viruses are the single most abundant biological entity on the planet. They're literally everywhere, in every ecosystem, every drop of water.
0:25They're modulating everything from, you know, our own health to global carbon cycles. We're absolutely crucial. And yet, when we try to study them at this massive scale, this field we call viromics, we just hit a wall, a huge bottleneck.
0:40And it's one of the biggest in modern biology. It's not about our ability to sequence anymore, is it? We can get the data. We can generate terabytes of it. Oh, exactly. The problem is interpretation. It's making sense of it all.
0:50Viral evolution is just it's relentless. Their genomes, their proteins, they mutate and diverge so fast that our traditional tools just can't keep up. So the tools that rely on finding a close match in a database?
1:02They fall silent. If a virus is too distantly related to anything we've ever seen and annotated, it becomes what we call dark matter. We had the sequence. We know it's there, but we have absolutely no idea what it does.
1:14No idea about its function. It's structure. Nothing. It's a black box. So if that sequence similarity gets erased by evolution so quickly, we need a totally new way of looking at this, a kind of genetic rosetta stone, a system that doesn't need perfect letter for letter matches, but can somehow understand the deeper grammar, the underlying structure of a viral genome, a shortcut to understanding these evolutionary chameleons.
1:42And that is precisely our mission today. We're going to unpack a really revolutionary genome language model that's built to do exactly that. To interpret this hyper diverse viral data, and really to build a new foundation for viromics.
1:55For this work today, we're celebrating the research of Cody Martin, Anthony Getter, and Carthic Ananthropan, who are primarily from the University of Wisconsin, Madison, and affiliated institutions. Yeah, their work is a fantastically Ford.
2:06It really demonstrates how deep learning can be applied to interpret this incredibly complex, diverse genomic data. It's a direct shot at that dark matter problem we were just talking about. So let's get a bit more context.
2:17Viromix, this large scale study of viral communities is a huge research area, but this diversity gap is, it's profound. It is. I mean, if you look at bacterial genomes, Usually have some universal genes you can use for comparison.
2:32But for varuses. Most of the data lacks any usable functional labels or those shared markers we need to build a family tree, a phylogeny. And that's where language models started to look so promising. Exactly.
2:44We have these incredible tools, protein language models or PLMs that most established one is probably ESM 2. And it's been trained on 1000000s of individual protein sequences. So it's amazing at understanding a single isolated protein.
2:57Phenomenal. It can learn its biochemistry is function, its structure, but that's also where previous attempts to use these models in firearmics fell a bit short. They were too focused on single tasks. Like just annotating one protein at a time or trying to predict a host?
3:11Right. They miss the forest for the trees. A viral genome is not just a random bag of proteins. Its parts are organized. That organization is a product of its evolution. You have functional cassettes, you know, all the structural genes might be clustered together.
3:25Okay, so there's a logic to the layout. A deep evolutionary logic. And previous models just didn't capture that. A truly useful foundation model has to understand the context of each protein within the entire genome.
3:37All right, so let's unpack their solution then. The protein set transformer, or PST. How does its architecture get at this idea of genomic context? The core idea is actually pretty elegant. PST models a genome, not as a string of letters, but as a set of proteins.
3:53A set, okay. Pick at the genome as a toolbox. PST doesn't really care about the order you lay the tools out on the workbench, but it cares very, very deeply about which tools are in the box. What each tool does in relation to all the others.
4:06That makes sense. It moves beyond that strict linear sequence idea. So how does the model actually process this set of tools? It uses an encoder decoder setup. It's designed to produce context to wear information on 2 levels.
4:20The individual proteins, and then the genome as a whole. It starts with the input stage. It takes those really high quality vector embeddings from a pre-trained model like ESM2 for every single protein in the set.
4:32So it's already starting with a rich description of each protein's basic features. What does PST add to that to make it context to wear? This is the really clever part. It concatenates, it just sticks on.
4:43These small learnable vectors to each of those big ESM2 embeddings. And these small vectors carry what information? Just 2 things. The protein's position in the genome and which coding strand it's on. That's how the model is forced, right from the start to pay attention to the genomes organization.
5:01Huh. That's fascinating. It's like you have a great dictionary that's ESM 2. And then you add grammar and syntax rules to every single word before you even try to read the sentence. That's a perfect analogy.
5:12And that enriched input, then goes into the encoder. It uses multi-head attention, and this is where the magic really starts. The encoder looks at every protein and figures out how it relates to every other protein in that same genome.
5:25So it's building the context. It's building the context. The output is a new set of embeddings. PSB protein embeddings that now encode both the protein's own features and its specific role within that genome.
5:37And then the decoder pulls it all together. The decoder uses a multi-head attention pooling mechanism. It's basically a very sophisticated waiting system that the model learns. It decides the relative importance of each of those contextualized proteins to create one single consolidated representation for the entire genome.
5:55And you get both the protein level and the genome level view in one go. In a single pass. Okay, so the architecture makes sense, but how did they train it? They didn't use the standard method, mask language modeling, they chose something called triplet loss, or PSTTL.
6:10Why that change? Well, masked language modeling is great if you want to predict a missing word in a sentence. But for genomics, especially viromics, our main goal isn't really prediction, it's mapping evolutionary relationships.
6:21Ah. And for these complex high-dimensional vectors, triplet loss is just far better because it directly trains the model to understand relatedness. So what does that look like? What's the triplet? Think of it like a family tree challenge for the model.
6:36You give it 3 things. An anchor genome, which is your starting point, then a positive example, its closest relative, and a negative example, some distant unrelated genome. Got it. The whole goal of triblet loss is spatial.
6:50It just trains the model to pull the anchor and the positive closer together in this big mathematical space and to push the -one further away. by guaranteed margin. So it's forcing the model to create a map that accurately reflects evolutionary distance.
7:04But how do they know which genome is the positive one to begin with, especially with all this unannotated data? They use a clever metric called chamfer distance. Without getting too technical. It's a way of measuring the difference between 2 sets of things.
7:18It finds the best way to match up the proteins from one genome to the proteins in another and measures the average distance. And make sure the positive example really has the most similar collection of proteins overall.
7:30And to make the model even more robust, they threw in something called point swap. What does that do? It's a form of data augmentation that mimics a real biological process homologous recombination, which happens all the time in viruses.
7:43So it's like genes getting swapped between related viruses. Exactly. Point swap simulates this by swapping similar protein vectors between 2 related genomes in the training data. By showing the model these slightly jumbled but still biologically plausible examples, you prevent it from just memorizing the input.
8:00It has to learn the essential features. And the scale of this was just enormous. Oh, massive. The foundation models were pre-trained on over 100,000 high quality viral genomes. That's more than 6000000 proteins.
8:12So let's get to the findings because this is where all that architectural and training sophistication really shines. How did PSTTL actually do when they benchmarked it? It was a decisive win. Across the board, PSTTL significantly outperformed every other method they tested it against.
8:29And that includes what? Everything from, you know, standard nucleotide camer approaches to other big genome models like Genes LM and high-ended DNA. And crucially, it blew simpler protein-based methods out of the water, like just taking the average of all the ESM 2 embeddings.
8:47That last point seems really important. The approach they called PSTCTX just averaging the vectors that failed. It performed very poorly, which is strong proof that the whole architecture, the context building encoder, and that learned waiting in the decoder is absolutely essential.
9:01You can't just throw the proteins in a bag and average them. The structure is vital. It is, but the most revealing result, the real aha moment came from its ability to detect remote evolutionary relationships.
9:12Okay, let's define that for everyone listening. What's the difference between a close relationship and a remote one in baromics? Right. So we use 2 key metrics. For close relatives, we use average amino acid identity or AAI.
9:24You can think of it like checking if 2 books use a lot of the same words. If the AAI is high, they're obviously related. And the remote ones. That's where the virus has evolved so much that the AAI drops to basically zero.
9:35The sequence similarity is gone, the words are all different now. But the books might still have the exact same plot structure, the same character type. So the underlying structure is conserved. Exactly.
9:45And we can measure that with something called average structural identity or ASI, which compares the predicted 3D folds of the proteins. So when AI is zero, Traditional tools are completely blind. What did PSTTL see?
9:59This is the amazing part. PSTTL showed a strong positive correlation with the structural similarity? With ASI, even when the sequence similarity, the AI was totally gone? Wow. The model is inferring this deep, conserved structural relationship, just from the genomic context.
10:16It's completely bypassing the need for sequence data. That's, That's groundbreaking. It's seeing the shared blueprint when the building materials look completely different. Did this translate to function too?
10:27Could it map out the organization within a single genome? It did. It was clear proof that the context to where training worked. PST consistently grouped related protein functions into what the authors call functional modules.
10:40Any specific examples? Yeah, the standout one was how it clustered the proteins involved in what are called late genes. So things like the tail proteins, the head in packaging proteins, the license proteins that burst the cell open.
10:52And why is that particular group significant? Because that clustering reflects a known, conserved biological reality. It's a common organizational pattern in viruses, like the famous Lambda phage genome.
11:04And the key thing is the simpler models, like average ESM 2, completely failed to see this pattern. It proves PST is actually learning the functional architecture of the genome. Which brings us to the biggest challenge of all that dark matter.
11:18The 70 to 90% of viral proteins with no known function. What did PST do there? This is where there's massive hope. The researchers noticed that these proteins of unknown function, these hypothetical proteins were often given a very high weight by the model's decoder.
11:34So the model thought they were really important for defining what that virus was. Right. And when they looked closer, they found that PSTTL was incredible at clustering these unknown proteins together with known capsid proteins that shared structural homology.
11:49So let me get this straight. Even if a hypothetical protein had no sequence matched to anything. If it had the 3D fold of a capsid protein, PST could group it into the capsid functional module. That's exactly it.
12:02It's transferring annotation based on inferred structure and context, not just sequence. This could dramatically expand our ability to assign function to this vast unannotated part of the viral world. Okay, so let's talk real world application.
12:14They tested this on a classic Miramics task, host prediction. They did. As a proof of concept. They took the PSTTL genome embeddings and fed them into an existing graph-based prediction framework. The results were astounding.
12:27Better than the current tools. Significantly better. It outperformed established specialized host prediction tools like iParsop at finding the true host species for the test viruses. What's so amazing about that is that TST was never specifically trained to do host prediction.
12:43Exactly. It speaks volumes about the quality of the embeddings it produces. It was trained for general representation. The fact that it excels at this downstream task means it's capturing truly fundamental biological information about the virus.
12:57The evolutionary map it creates inherently contains the ecological map. So what are the bigger implications here? Where does this research go next? It wasn't just built for viruses, right? No, not at all.
13:08The architecture is totally agnostic. You can feed it any set of proteins. The authors are very clear that it could be readily applied to create a foundation model for all of microbial genomics bacteria archaea.
13:19Which face a lot of the same problems with divergence and annotation gaps. For sure. This could really revolutionize how we annotate genes across the entire microbial tree of life. What about the limitations?
13:30What did the authors say needs to be improved? They were very transparent about that. One clear path for improvement is that PST uses fixed embeddings from ESM2. They think performance could be boosted even more by fine-tuning that input PLM at the same time as a main PST model.
13:47So training the whole system end to end. Right. And another idea was to use a dual training objective. One that focuses on protein, protein relationships, and another on genome genome relationships to get an even finer grain map.
14:00And before we get to our take-home message, they did include a section on biosecurity. With a powerful viral model like this, what was their assessment of the risk? They did a full assessment and concluded the risk is low.
14:11The main reason is that PST works at the level of a set of proteins. That makes it really difficult to use it for, you know, de novo generation of a full functional viral genome. Plus, the training date itself contained a tiny fraction of human pathogens, less than 0.2%.
14:26They also consulted external experts who agree that the huge scientific benefit of releasing the model in code far outweighs any low theoretical risk. This is truly foundational work then. The protein set transformer, PST, really seems to solve this dark matter problem in viromics by moving beyond simple sequence matching.
14:46It absolutely does. It proves that by intelligently processing the genomic context of these protein sets, where they are, how they're oriented, which ones are important, PST creates a far superior foundation model for understanding viral evolution and function.
15:00So the central insight is that in these hyper diverse genetic spaces, contextual awareness is really the key. It's everything. It's the key to an accurate genomic interpretation. So that leaves us with a final thought.
15:12Since PST proves so effective at accurately linking these incredibly diverse viruses to their hosts, based purely on this conserved functional organization, how might this new context-ware approach fundamentally change how we design targeted interventions like phage therapies for diseases related to the microbiome?
15:31This episode was based on an open access article under the CCBY 4.0 license. You can find a direct link to the paper and the license in our episode description. If you enjoyed this, follow or subscribe in your podcast app and leave a 5 star rating.
15:44If you'd like to support our work, use the donation link in the description. Now stay with us for an original track created, especially for this episode, and inspired by the article you've just heard about.
15:53Thanks for listening and join us next time as we explore more science, base by base.
