Law firm review layers change AI’s minimum quality bar

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 29:53

   I think if you look at where AI has worked really well, it's where, you know, where hallucinations are a feature, like in the creative industries, right? Hallucinations are a feature. And your industry i'm assuming hallucinations are an absolute bug and so what did you do to make you know those argumentative lawyers really trust yeah really trust what the model was saying so i'd actually push back on the second piece a little bit okay in a lot of what lawyers do is creative actually like a lot of it um so like i'll give you an example for this litigation would not exist if there were just 10 really simple rules and everyone knew which fact pattern fell into the boundaries of those rules like there'd be no litigation right and so i was a litigator i mean transactional is also incredibly creative but litigation i think is just a better example

2. 30:37

   because what you're trying to do is i have these facts here are somewhat of the rules of the game let's figure it out right and that's actually incredibly creative like very creative um so but still who like accuracy is more important than creativity to be clear here um and so on the accuracy piece so even given that creativity is actually super useful in some instances and we try to make it so that our product doesn't like get rid of that aspect um on the accuracy side there are a lot of things you do to improve it for one um but for two the main thing is law firm are hierarchical. So you, like a junior associate, if they get a task, they do the first draft of it, and then a second year reviews that, and then a fifth year might review that, and then the partner reviews that, and it goes out, right? And so actually the, you know, minimal viable quality

3. 31:32

   of your output can be lower for a law firm than it can be for an in-house team, actually. So selling to the law firms was also helpful in the beginning because so much of the work gets reviewed, right and so you aren't selling them something where they kind of click it and forget it right they're actually in the loop at all times when you're selling to an in-house team it is better to sell them the specialized versions of tasks that they can kind of see an insanely high accuracy level on and the more specialized you are building the higher accuracy you can get because it's easier to fine-tune it's easier to evaluation because there are less steps and the service area is just smaller. Let's talk about the lawyer of the future and the law firm of the future. Foundation models