Biology models need lab-based verification loops

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 42:24

   So in the context of language modeling, there was a time, and we've kind of grown out of it now, where the overwhelming amount of compute was just going into simple pre-training on data of, like, kind of unknown quality and relevance. And then you do a little, like, instruction, you know, following at the end, and all of a sudden you have, like, a pretty helpful assistant. What do you think that recipe is going to look like in biology?

2. 42:51

   Like, can we just take tons and tons of raw, but kind of not that great data and then use that to build up these representations that we then sort of elicit what we want at the end? Or is it going to be sort of a different path that you think for for whatever reason? Yeah, that's that's also an amazing question. I really think that the principles in biology are going to be shared with all the other domains where we're seeing progress in AI.

3. 43:18

   And so I actually think, you know, we're fortunate in biology because we have kind of the opportunity to see how those principles have played out in different fields and really identify the parts there that are generalizable. But I think this core idea of the power of self-supervision or representation learning, you know, the pre-training, you know, that's really critical because, you know, that gets you beyond.

4. 43:43

   And, you know, this was... dream of Hinton and Jan LeCun and Bingio and kind of the other deep learning pioneers is the idea that you didn't have to know all the relevant variables. You didn't have to encode all of them and know the settings for the parameters that, you know, you could learn all of that with a neural network. And so that's just, you know, proving out over decades. And the scaling laws are really, I think, the latest or the most clear expression of that now. But so we have pre-training in biology. Certainly, we're seeing that playing out with proteins right now, of course. And I think that will play out for cellular biology too. And I think it's going to be similar. You know, there's a phase of really learning from statistical patterns to build a model of the underlying system that's driving those statistical patterns. And then the stage

5. 44:48

   of elicitation where you're really now bringing out the part of that model, which is relevant to the problem that you're trying to solve. And I think that it's going to be possible to have rlvr for biology and to have feedback and you know ultimately where where we need to get is to really build empirical ai and kind of build systems where where ai can can you know ask the question perform the experiment learn from the experiment improve its understanding propose a better experiment and really complete that that cycle repeatedly so that you have a recursive improvement it. And I think, I think all of that's going to be possible in biology. It's, it just has, it's going to need to be built differently. You know, we're going to need, it's going to need to work with models that have, you know, that can, that can accurately represent the, the

6. 45:37

   multi-scale nature and the complexity of, of biology in the relevant ways. And, you know, the biggest challenge for biology is it's, it's empirical, right? And so, you know, we're seeing so much rapid progress with our LVR in areas, you know, where we can really do fast verification computationally. So, you know, mathematics, coding, you know, reasoning and things, things like that with biology, you know, we have to build the systems that actually, you know, allow us,