Harvey raised its legal benchmark score from 4% to 18%

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 4:43

   through the process yeah so up until now it's been really hard to to just go from this fact of like data to intelligence up until now that's only been in the labs uh and you need a massive research team uh when uh so in the last couple years we We started to have vertical companies going into this space as well and building a massive research team. So I was previously at Windsurf where we embarked on this journey and trained a model.

2. 5:10

   So we won. So did Cursor as well with Cursor Composer. But the problem was you needed hundreds of millions of dollars of R&D in compute. You needed a really massive research team to invest in. And then you also needed a lot of time. So it took us six to nine months to actually get to that point where we could train a frontier model. and that is with a kind of hand-holding sort of consulting model that's still sort of the world that we live in I think any good product exists when you have every single company in the world trying to do the same thing over and over and over again and if you can package that into a really beautiful interface that just says here are the knobs and levers that you care about to translate that what good looks like and then everything else is packaged away I think that's that's a

3. 5:54

   beautiful interface to do that. What did you and your co-founders see when you were at Winsurf and when they were working on models as well to help you realize, hey, this is repeatable, that there is a product to be built? When did you have that aha moment and that realization? Yeah, so part of it was, you know, we were training these models on production, like basically saying, what are all these edits, the retries that are happening in the product and learning from that at Winsurf?

4. 6:19

   And then combined with that, so my co-founders, Michael, was at DeepBind as well. He was working on Gemini Robotics. And they're starting to see the same thing, where you could actually deploy these robots in the real world, build continual learning infrastructure to basically allow these robots to update every single day. And then Arjun was at Apple working on the Vision Pro. And the same thing was he was basically saying, how can we take artist feedback and use that for multimodal foundation models?

5. 6:46

   So in one way or another, all of us were basically figuring out this grand alignment problem of how do you actually take this feedback from the real world and improve the models? The problem is we were all at these big research labs, right? And so I started to realize, okay, continual learning is going to be the next learning paradigm. Is there a way that I can actually do this within the labs? And the thing I very quickly realized is at DeepMind, our entire focus was training one massive model and then distributing that one model to everyone in the world.

6. 7:16

   And that's been a winning business model, obviously, across the board. But if you want to basically take one model, customize that for a single person, and then distribute that one model just for that customer. Like, for example... what our platform does with Harvey, that is an immense amount of infrastructure for a massive lab to build just for one customer. Yeah, talk about what you guys have done with Harvey, maybe as an example.

7. 7:39

   So Harvey, basically, they were up until now riding on top of the frontier closed source models, right? And they've been able to build incredible interfaces and products on top of that that help tons of law firms in the world. But what they realized is they have an immense amount of knowledge, of data, of all of this nuance built up that they are actually able to see what the law firms actually care about and the built-up experience of just 30 years of working in this domain. And with using frontier models, they're unable to actually take that data and use that to their advantage. And so what they realized is, hey, okay, they're able to construct, for example, a legal agent benchmark and what they released and using all of the expert information that they have. And then if they get the power to basically

8. 8:25

   use that data to train a model, then they get to shape their own destiny. So with Trajectory, what they were able to do is train a model that is six times cheaper. And it goes from around 4% accuracy for an open source model to around 18%. And there's still so much room to grow, obviously, what they're able to do, but now suddenly get the power of a research lab. And on top of that, the nice part is these models end up being six times cheaper than the Frontier. And so you're able to expend so many more tokens to achieve these tasks.

9. 8:55

   So you picked the name Trajectory, as it's related to this Continue to Learn concept.