Early reasoning models solved hard diagnostic cases doctors often miss

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 40:04

   And you helped OpenAI start up their healthcare platform. How did that get involved? So, yeah, so our time with OpenAI was, my gosh, I guess it was two and change years ago. We were talking with Brad and Sam about healthcare kind of writ large. And this push that, look, they should spend more time with more of the leaders in healthcare because this is going to be a key space for them. So therefore, they should spend time with the incumbents both to understand how do they work with them and to understand the space better.

2. 40:34

   So organize a series of meetings with different leaders across healthcare. So it was some of the top biotech CEOs, some of the top payer CEOs, and some of the top health system CEOs as separate kind of events with the open AI leadership to actually just sort of talk through and say, here's the state of the art in AI. Here are these folks' problems, and let's talk through. I remember with the health system leaders at the time, the O1 wasn't out yet.

3. 40:59

   So reasoning was a brand new thing. It hadn't happened yet. But because we were close to them, I'd be able to play with the reasoning models myself. and I'd started putting the New England Journal of Medicine case studies. So that's, think of the house MD level difficulty hard cases. I would put the first half of them in because the way they tell the story is, you know, here's their history, here's this, here's that, and then something happened.

4. 41:20

   And then they'll tell you about what the expert said and what it was. So I'd cut it off right there at the end and then it happened. So I just put that portion in every week. And I was just blown away by the fact that, you know, every single week they would get it right. And these are cases that neither me nor my colleagues, or some of the best doctors out there, maybe we would get it right two times out of 10.

5. 41:42

   And they were open AI and the 01 was batting a 10 out of 10, which was amazing to me. So I remember going to that meeting and saying, look, I think these models are amazing. These models can diagnose. And everyone was treating me like I was a heretic. It was like, this is blasphemy. What do you mean? This is impossible. And I remember we then showed them. and so did a couple of them live. And just seeing the eyes of some of these healthcare CEOs, some of whom were also doctors, it was just amazing starting to see it click for them that, oh, wow, this is the capabilities of these tools.

6. 42:13

   This is what's happening. This is what's coming down the pipeline. Same with one of the biotech CEOs. We were like, okay, what do you think of these tools? Let's talk through them. And they're like, well, I think they can be really good for customer service. I don't really think, though, these tools will ever be useful in drug discovery. And now I'm seeing two years later, One of those same CEOs has announced billions of dollars of spend in this.

7. 42:34

   So it's been also very interesting seeing the progression of folks' belief in the tools in such a short amount of time. That's a really good point. So how has the tool stack changed for these?