Protein models can connect unknown proteins to known biology

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 14:22

   One of the things that inspires me most about this effort is really what Priscilla said, which is like, well, there's so much we actually don't understand about biology. And what if we could, which I think is actually very different from lots of other incredibly interesting and useful AI problems we attack. We're like trying to replicate human behavior. And I'm like, a lot of that data is, you know, on the Internet or captured and without pretending to understand all human behavior, you can predict a lot of it.

2. 14:46

   I thought one of the most interesting things in your release was actually, you know, the like mechanistic interpretability stuff you alluded to, which is, can we actually extract new knowledge from, you know, what the model believes? happening, right? Can you talk a little bit about that? Yeah, I'm really excited about that. So I think, you know, in mechanistic interpretability, kind of traditionally, it's been applied to large language models with the goal of understanding, you know, kind of what is the representation space of a large language model? How does it compute things? And does that really connect to, you know, what we understand about our intuitive understanding of the world? And so there's, I think, this really rich toolkit that has been developed to start to be able to ask those questions. So kind of what does that mean for biology? One of the classes of models that

3. 15:36

   we train are these protein language models. So they're really, you know, it's trained on the codes of proteins. And so anything they learn about biology is kind of emergent. And we've seen that they can learn things like biological structure and biological function. And that's just kind of emergent from this, you know, token prediction training task. So, you know, as we think about like mechanistic interpretability in those models, you know, we're really seeing the unknown because the models have been trained on billions of protein sequences. They've been trained on, you know, both known and unknown biology. And yet they're developing these representations that start to kind of capture things that we can really see correspond to that reductive picture of biology that's been built up over the centuries. So kind of you can start to

4. 16:21

   connect the dots between proteins where we kind of really don't know anything about them with proteins where we do know something because there's that kind of underlying structure grammar that's linking them in the representation space of the model. And at the extreme, it could be, we're going to understand systems in the body that we didn't before or the mechanism of action for a new treatment because we can ask the model, right, interrogate that representation.

5. 16:48

   That's right. The hope is that you kind of really learn the underlying basis for how it's making the predictions. And so you open up the black box and you can actually understand kind of the