Legora uses customer evals to decide when a legal task is solved

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 23:05

   aware of i mean on roadmap like way back every new model just like unlocked new things right and when we got early access to gpd like 4.5 and you just realize that holy shit like now it can finally draft like an end-to-end thing and we don't need like all these harnesses and like things around it that's amazing like let's unleash it in a way that that works by the way to do that you need sort of like a low ego organization because you build all this ip and all this software yeah and you're like okay now the model can do it yeah delete it all worked really hard for six months yeah we're deleting everything yeah it's incredible yeah but i think a lot of the things that we have built we know that we're gonna delete someday and i guess you kind of people to opt into that at the front end for that culture to really work.

2. 23:51

   We've also talked about it as like, if we were, if we're here today and we start building for the future that's over here, like that's too far out. Like... our customers are not going to adopt that. They don't understand it yet. So we need to take them on the journey and we need to take them on the path of being successful, which I think every iteration cycle now is shorter. Like back in 2023, 2024, I think it was like slightly longer.

3. 24:14

   Like you'd have a quarter or two quarters because the models weren't moving that fast. Every upgrade was pretty incremental. But now it's like flipped. Opus 4.6 flipped in capabilities. So now we have to revisit a lot of the things that we built do you know what the next flip you're waiting for is like is there a thing so actually i don't think so it was funny i was at the customer advisory board at anthropic yesterday which is i'm wearing my like dario shirt here you look like dario thank you most of that conversation was about the models are now intelligent enough where they're no longer the bottleneck the bottleneck is all of the software around putting the models in an environment where they can execute and do work and humans can review that work in like a trustworthy way.

4. 24:59

   They're seeing that across basically every single vertical and every single company. So I don't really think that we're waiting for new model capabilities anymore. There's like nice things to have. Like it's nice to have better context windows. It allows us to do less garbage and context management or when you overflow the context in memory and so on, you have to like deal with it to refresh it. So there's nice to haves, But I think we're at a point now where we just have so much building in front of us in terms of bringing the model capabilities into our world that that's where all of our focus is.

5. 25:32

   I think on sort of discovering what the models can do, we thought very early on that evals were going to be important and both building up like an exercise of building new evals, but also building out evals for all the use cases that we want to cover for. Because in the beginning, it was a lot of like, oh, how good is Sonnet? How good is Gemini? How good is GPT? And so we had to test them on the different evals.

6. 25:53

   And a lot of our customers actually contributed this. So they would give us manual tasks that they used to do. And they'd tell us, here's the evals. And we're going to call you when we can get to 100% on these evals. And I actually remember, it was a funds-related use case, an LPA key term review report that a danish law firm was spending like three three days on basically like an associate would spend three days putting together that report in summer of 2024 we had like 60 accuracy on that task by the end of that summer we had 100 accuracy and once you got to 100 accuracy i mean that task is done like it's it's over i've adopted this mentality internally that if ai can do something it will do it and so our product like we think a lot about solving legal tasks end to end and once a task is conquered it's done like we just like strike it out and we're on this path of solving

7. 26:58

   more and more complex tasks like you start with NDAs but at some point you get to full-on share purchase agreements which are very complex but we're gonna get there I think the question for these organizations who are maybe more traditional and trying to keep up with the pace of AI is how