Client confidentiality forces Legora to infer product failures

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 26:18

   How do you know that we're actually doing something good here? How do you think about that? I'm assuming that's also related to evals. So we have the regular product metrics that we care a lot about. Are people spending a lot of time in our product? Do they seem to get a lot of value out of it? Do they renew it? Do they come back a lot? We have really good numbers for an enterprise software of our Dow over Mao and our hours spent per month in the product are extremely high.

2. 26:45

   And then we do online evals also, where it's like, try to figure out, generally, did we solve the problem here? Just with a super cheap LLM message, I just try to just have an online metric, and then we can use that also when we do rollouts of new things. We can just compare feature flag-wise, okay, did it improve or did it catastrophically decline? Because one thing actually that might be interesting that's difficult for our business is we can't look at any traces basically.

3. 27:08

   We can't look at any data because it's all super confidential. And that's like extremely frustrating to work with because you basically don't know what people are doing. So it's like, here's an agent and it can do so many amazing things and I have no idea what people are actually doing. I would have assumed you guys like signed some sort of agreement with the law firm and they're like, oh yeah, I wish.

4. 27:25

   Yeah, I wish. No, but the thing is like a lot of the data that they have is not even theirs, right? It's their clients and it's super confidential. So we can't look at it. And so basically we have an evals team that's like trying to approximate what we think everyone is doing and build that into eval sets. So that's just a super annoying kink of the business. Can you do paraphrase? So for example, ChatGPT has a slash feedback thing that directly fully sets the full trajectory.

5. 27:49

   But obviously that is not possible for you guys. But you can use an element to paraphrase from this real thing, make a synthetic thing that is close enough but change all the names, change all the details. but like still yeah we can't I mean there's always a range right of like if you if you paraphrase all the way down to like three values and you pick one of them then obviously that's fine if you just remove the names then that's maybe not fine yeah it's like we are generally pick the most safe way okay so like we restrict ourselves as most possible because we you know there should be no risk at all yeah but there are things where it's like okay we know that the failure modes are probably these 10 or 20 so we can classified into one of those and that's a helpful signal. We can classify it into the

6. 28:35

   types of tasks and the practice areas also and then we have an idea. It's something that we're always trying to figure out how to do in a smart way. Also on the privacy front, I guess, I'm assuming even two years ago it must have been insanely difficult to get the law firms to be