A four-billion-measurement contest found a generalization gap

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 28:21

   Okay, so this Kaggle competition, I think, is just a fascinating story because you had such high hopes going into this. We did. And I think this is one of those cases of very nice epic rigor where you didn't just provide, oh, here's a train test with a random split or something. You thought carefully about how to organize this Kaggle competition, how to run it, and how to kind of evaluate fairly. Can you talk about that process, the decisions you made, and then also the results?

2. 28:54

   Yes, happy to do that. So we were not by a long shot the first group to try to evaluate whether you could use computers to make reasonable predictions with small molecules binding to chemicals and then test those predictions. There are lots of papers that are doing this. There's a real classic paper that came out of Xchem and Google some years before we found it Leash that does this. we had run a lot of our own experiments on public data and we had suspected that no one was really very good at it.

3. 29:22

   And so we wanted to put together this contest and design the evals in particular in a way that could really test it across a range of eval difficulties because we were thinking about, okay, we're going to have a lot of people, hopefully a lot of people compete in this. Some are going to be very good. Some are probably going to be beginners. We want to make sure that there are some nice, easy wins and like a bunch of stuff in the middle that's like kind of hard and then some things at the very top end of what would be difficulty or what we imagine difficulty might be.

4. 29:50

   And so to kind of unpack that, we've talked a little bit already about how these libraries are combinatorial. So you're going to have some smaller number of building blocks, but a large number of combinations. We got our hands on a chemical library from a wonderful sponsor, this guy named Xiaojie Liu, who runs a shop called AlphaMah in Shanghai. And the chemical library was about 130 million molecules.

5. 30:10

   and the center part of the molecule, you can imagine it kind of like a hub with three spokes and then the center hub was the same for every molecule. It's a particular chemical motif called a triazine. And so what we did was we said, okay, so we got the center hub, then we got these three spokes. For the evaluation data set, let's have a couple different ways to split this. One way is that we could say, all right, the chemicals might have one shared building block, one shared spoke, and then the other two spokes are different.

6. 30:41

   Okay, that's one thing we can do. So let's do that for some. And that was the easy stuff. Another one we can say is, all right, well, maybe we're not going to have any shared building blocks, but we do know that if they're going to come from this chemical library, there's going to be the shared hub. And so we can imagine that some of the spokes might be similar, but they won't be exactly the same and they won't be shared.

7. 30:58

   So we think this is going to be a little bit harder. Sounds great. And then for the third most difficult eval, we had a completely different chemical library that we had designed at Leash. There was no common hub, no common building blocks, different attachment chemistries, different everything. And so we thought, okay, we're going to do, we're going to have these three in different proportions. We're not going to tell anyone that's what we did. We're just going to say, here are the evals, and let's go see how it goes. And so we launched the competition. It was two years ago, and you guys can look it up if you want. We've written a lot about it. And then after 2,000 teams competed, the winner did okay on the stuff where they were sharing building blocks.

8. 31:36

   Depending on the target, we saw some boosts performance for where there were no share building blocks, but the core hub was still the same, the TriZine was still there. And then when you go to the most challenging eval, no one did better than chance. Yeah, the winner was the 1,200 just random person who submitted a Jupyter notebook, right? So the winner was this guy and Viktor Shlopov who put together a slightly different approach, and his was good, but again, like everyone else, no better than chance.

9. 32:01

   Three of the top five were the noisy tutorial notebook. But just to be clear, usually with Kaggle competition, there's like a public eval for you to train to test your algorithm, I guess, and then there's the private one that's held out. That's right. And the competition runs that, and all you know is the score. Yep. So, was your library included in the public eval? No. It was hidden back to make it spicy. Right.

10. 32:28

    Okay. And it was spicy. It was very spicy. So, what did you take from this? What was the lesson? And I mean, first of all, what did you hope to get from this? You kind of touched about that, but then how did that sort of affect your thinking about, like, how do you proceed forward? Yeah, yeah. We had a couple of hopes, and some were fears. One thing I will be candid about is, you know, we had released this data set that was a thousand times bigger than anything anyone had ever seen, and we gave it about a 5% chance that recent advancements and kind of like co-folding approaches would mean that it would obviate our company entirely. Didn't happen.

11. 33:02

    No, didn't happen. No, but we were like, you know, there's like a small chance, like, we are really messing up here. And that didn't happen. We suspected more likely that folks weren't very good at it and that the data might help. And so we were hoping to see that. And anyway, you'd want to know that. You'd want to know that. Absolutely. And like another thing that we wanted to know is there's lots of different ways to represent molecules and give them to computers.

12. 33:28

    You could use these things called smiles that do it some way. You could do 3D graphs. There's like weird spherical convolutional nets. People have tried kind of goofy things to try to represent molecules in a useful way. And there'll be lots of people in the field who will argue that, like, oh, I think this way is better than this way or whatever. And we did really encourage competitors, especially those who had a chemistry background, like, let's try some of these different representations, so let's see how it goes. And we were grateful to find that the thing that we were doing, which is just basic smiles, kind of beat everything else.

13. 33:57

    It wasn't like a resounding thumping, but it was better enough for us to continue to using that approach. And smiles just if I recall correctly, is just basically a sequence, a textual sequence of strings. It doesn't encode any 3D structure in principle. It does not. It's all emergent. It is a graph traversal translated into a text string. And so you can start the graph at different points. And so there's what they call canonical and non-canonical smiles, depending on where you start. But yeah, it's an extremely simple representation. And so yeah, we were thinking that, or we wondered if some of these fancier ways of representing molecules might beat at least this particular problem and they super didn't so was your takeaway that you should go full force into this was this encouraging to you

14. 34:40

    or was this discouraging you that do like how what was the what was the collective emotional reaction at least after that i always i've always wondered that i mean he kind of wrote about you had several follow-ups um but i i'm just curious to hear yeah yeah this is a great question um so like we we are extremely bitter lesson pilled over here. And so our thought was some of it was kind of delight, which is like, wow, this is much harder than we thought.

15. 35:06

    That's kind of great. That's kind of great. Yeah. Because if this problem is like all the other ones that we're aware of, if you can get pretty good signal out of a scalable measurement regime, then you should just try to go get that. Yeah. Yeah. No, I mean, for us, it was pretty validating, like, oh, man, like we, this is an opportunity here. Especially if your scalable measurement regime is hard and you have specialized knowledge to do that.

16. 35:30

    Absolutely. Absolutely. It was like, oh man, this is the time to pour gas on this. This is great.