Most model calls should serve software, not people

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 1:23

   I think actually, in many ways, even more interesting that were some weeks removed from it. Given that the early results were not what you expected and maybe not as strong as you were hoping, what gave you that kind of early confidence in this research direction of RLCD? I mean, I feel like you kind of alluded to it in the GPT-3 example. It seems like there has to be a leap of faith that any great research effort undergoes. And what was it about this approach that just in the early days when you didn't have the signals early on, you were still kind of confident in? Oh, man. Oh, man. Oh, man.

2. 1:55

   this is like i'm getting a little emotional right now um i have a fun little uh side topic that gets into this if that's okay my co-founder was debating on whether or not i was mission driven at open ai i would have described myself in hindsight as one of the lowest percentile mission driven because i think that the open ai mission is not really um it's kind of intentionally left as vague so many people's missions can fit into it but like the grand mission was never something i super believed in even though i truly want ai to be extremely extremely transformative so i'm still trying to figure out like am i was i mission driven then or not because i'm really mission driven now but the answer was you know just a lot of work trying to square like ai being super incredibly awesome which i think it is and ai being really not that useful in every like

3. 2:46

   productivity stat around and this is like the kind of thing that like most people like just focus on one or the other and they never had like a bridge answer or like they never have a sensible bridge answer i think distribution is like a nonsensical answer here um because so many companies have gone all in on that and then gotten burned um it's it's almost certainly not a distribution and to me it was like believing that like hey ai is clearly smart and also that the future is clearly going to be machine native the question that always motivated me and like i used to ask this question even to early people joining or even to investors in our early, you know, like our early rounds or even like the precede round, which I don't even know if that counts, like taking a safe. I used to ask this question of like, OK, like you have imagine LLMs

4. 3:33

   are an API, you know, like a like a sub module that is called by software, which even in the most string model ends, they still need to be called by software by something like a harness. that harness can be a while loop. But what percentage of those calls going into the LLMs are actually meant for human consumption? So for consumption, so things like style matter, and, you know, pros and all of that, versus for computer consumption, which is going to be very, very different, you know.

5. 3:59

   Like, I don't think a computer really ever wants a string, except when that string happens to be an extreme cardinality enum, which is a very valid use case. But it's very rare that that is the thing that computers want because they operate on bits. And when I just ask myself or anyone this question, even the most string-filled of people will give many nines of, like, percent for the machine-native component.

6. 4:21

   It's just a question of when that will happen. And to me, it was always like, okay, what does that thing look like? let's work backwards from that cognitive core and make it work. And I will say JEV is not yet there. There's still so much more work to do to make it so much better. And that's what really drives us all. Is JEV not yet there? Because I mean, you know, today, the shape of like, you know, calls and tokens actually is more for humans than it is for computers themselves. How much of it is just you're also waiting for the use cases themselves to evolve? And how much of it is, you know, areas you want to push on the model i think it's a bit of both it's it's like this interactive dance and this is also why we released like these dreamer ready features like the things that we are you know like wouldn't recommend for the most stable of use cases because we really want to

7. 5:11

   be honest about our model's properties so we've released a lot of these things because the reality is that the people who best know how much reliability you know robustness like intelligence is needed for their downstream application is not us, it's the developers themselves. Because developers are the expert at this and they will know way better than anyone on our team. We have smart people, but they will know better what reliability profile they need as long as we give them the right knobs, which string models don't tend to do.

8. 5:37

   So there's an entire spectrum of this. I think that they need to co-evolve together. The reason why we launched was not like, okay, this is like a good moment in AI. Everyone told us not to launch then, just an FYI. It was that we felt like, hey, this is getting to be something we're really proud of like it's kind of like impressively smart at a lot of things if you prompt it right and we are ready to like keep pushing that limits that those limits with the rest of the field to help us co-evolve use cases figure out weaknesses deal with jaggedness like there's so much we do to focus on jaggedness and insufficiency and intelligence that having people