Legora tests agents in simulated legal workspaces

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 8:44

   Sub-agent delegation was one of them that we've gotten working really well now for fan-out type retrieval queries. It works extremely well for us now. That was difficult before. We basically track, so we have a team that's just doing evals, and so we track all of the model's performance in different practice areas, which is like subgroups of legal, and different tasks across all of those. So we basically have a big heat map, and we're like, and then we can see them move and it's like, usually they improve most stuff, but there's actually a lot of model releases where they improve some and regress in others.

2. 9:15

   Okay. And that actually happens a lot more than you'd think. Like, what are some of those tasks, like, are you familiar with how you guys are running a lot of the evals? Like, what are those different sub-segments, I guess, that you look at? So, like, there's different just, like, framing stuff. So, we have different basically, like, worlds, like, RL environments, essentially, which is, like, here's a because Laura's working projects always more or less.

3. 9:36

   So it's like, here's 10,000 file data room. You're doing a due diligence. In the end, a grid report looks like this. Like it contains all these red flags and like it doesn't contain whatever weird stuff. And like we want it to be concise, et cetera, et cetera. That's kind of how we've built it. So there's different tasks, which is like part of it is drafting a report. Like does the draft look good?

4. 9:56

   Does it contain all the right stuff? Is the wording sound, the analysis sound? There's also the part of finding the red flags, like the analysis before that. So you can kind of like self-divide it a little bit. And that's how we run the evals. Interesting. What does your evals team look like? Do you hire lawyers to do that? So yes, it's a mix. We have engineers and then we have lawyers. And it's one team.

5. 10:17

   So there's like three engineers and like three lawyers right now. And then we have a pool of what we call legal engineers, which is people that have worked in big law for like three or four years. Typically, maybe they have a CS degree and like they're very tech forward. and then they come to us because they want to help change the industry. Are they IT in big law? No, they're lawyers in big law. They're lawyers in big law.

6. 10:39

   With CS degrees. Yes. Oh my God. It's a Venn diagram. Yeah, exactly. That's like our hardest person to hire for. But they're also such a force multiplier because we can pull on them for the eval stuff and for the product development, which is like you have a pool of 100 experts that use the product every day and go and talk to clients every day. And also like they live this and have lived this. And we just pull on them to do evals and help us shape the product and it's such an advantage.

7. 11:03

   And curious, when did you guys start that team? Evals grew out of the assistant agent team at some point. I think there was a point up until recently where it was like, honestly, just not really worth it. Because all models just got better. So it was like, do we want to spend a lot of our time doing really complex evals or do we just want to build? Because there was so much to be gained from building a better harness and just sort of leaning into everything's going to improve and everything kind of just improved.

8. 11:29

   Yeah, like every two, three months, you get an infrastructure model. Yeah, exactly. Just pass it through, right? Exactly. I mean, we had one problem where it wouldn't site properly, and so it would be difficult to follow. And it was like, next model release solved. It went from like 70% to like 99%. It's just like we did nothing and like, great. Yeah. Maybe the real work is keeping all the data private,