# Legal AI needs process data and expert evaluation Markdown copy of https://realalpha.app/s/legal-ai-needs-process-data-and-expert-evaluation/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy. - Speaker: [Winston Weinberg](https://realalpha.app/builders/winston-weinberg/) · Winston Weinberg is cofounder and CEO of Harvey. · Co-founded and leads Harvey, after working as an associate in securities and antitrust litigation at O'Melveny. - Host: [Sonya Huang](https://realalpha.app/people/sonya-huang/) · Partner, Sequoia Capital · Works with OpenAI and Anthropic; co-wrote Sequoia's essay on generative AI. - Host: [Pat Grady](https://realalpha.app/people/pat-grady/) · Co-steward, Sequoia Capital · Joined Sequoia in 2007; backed Snowflake, Zoom, ServiceNow and OpenAI. - Show: Training Data - Episode: [How AI Breakout Harvey is Transforming Legal Services, with CEO Winston Weinberg](https://realalpha.app/v/eXK-_yyQDMM/), Mar 11, 2025 - Moment: 9:09 to 13:20 of the episode (4 min), playing on https://realalpha.app/s/legal-ai-needs-process-data-and-expert-evaluation/; the episode from there: https://realalpha.app/v/eXK-_yyQDMM/?t=549 ## Quote > “And then you just **chain the models on top** of that” > — [Winston Weinberg](https://realalpha.app/builders/winston-weinberg/) at [12:00](https://realalpha.app/v/eXK-_yyQDMM/?t=720) · Training Data ## Takeaways 1. Route requests based on the user's intent. 2. Retrieve the right internal and external context. 3. Experts supply process knowledge models lack. ## About this segment Winston Weinberg breaks workflows into intent, context, and checking whether the result is right. Because legal processes and standards like market terms aren't online, Harvey relies on domain experts and experienced evaluators. From “How AI Breakout Harvey is Transforming Legal Services, with CEO Winston Weinberg”, Mar 11, 2025. [Watch the full episode](https://realalpha.app/v/eXK-_yyQDMM/) ## Transcript Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there. [9:12](https://realalpha.app/v/eXK-_yyQDMM/?t=552) i would categorize this in three areas so the all you need to do for kind of like every single workflow is basically what does the user want what is the intent right Right. After that, it's so what is what do they want? What's the intent? How do you extract that intent out of them? The second piece is what context do we need? And the third piece is, is this right? Right. And so my point is, there are different systems that work really well for different versions of those patterns. Right. So routing, like using the models to actually like do predictions and routing is really helpful for the first one. Right. [9:45](https://realalpha.app/v/eXK-_yyQDMM/?t=585) And so that's the example of either follow-up questions is really important there. Routing someone's query to the particular task you think they want to do is very good. And so there's kind of like an orchestration element there. Context is, okay, do you have predefined systems that will search for your internal documents that are relevant to the question and then your external documents that are relevant to the question? [10:10](https://realalpha.app/v/eXK-_yyQDMM/?t=610) A lot of that work is retrieval, right? Like most of what you're building there is retrieval. and then routing to make sure that you are accessing the external documents when you need to and the internal documents when you need to as well. And then the third one is, is this right? Right. So going back to my citations example, which is silly, but it's actually really important. I think there also a lot of the work can be done by the models, but you have to make sure that the models are very good at checking for certain things. So let me give you an example of this. [10:42](https://realalpha.app/v/eXK-_yyQDMM/?t=642) there's a thing in legal that is what is market, right? And the models don't know what market is. And there's various versions of what market is. There is market for a particular private equity firm, like what terms they are used to doing on an LBO or a deal or an aside letter, et cetera. There are terms that are across all of private equity. And then there are just kind of like general M&A terms, right? And the models don't have access to this data, right? And so on that third piece, a lot of it is, can you build a system that is very good at retrieving all of those different data sets when needed and comparing them against each other? [11:21](https://realalpha.app/v/eXK-_yyQDMM/?t=681) And I'm sure it helps if you have all of those different folks as customers. Yes. Yeah. So, I mean, that I think goes to another piece, which is the biggest problems that we have, and I think, I mean, is a very interesting problem, and I don't think the models are just going to solve this is the process data for a lot of these tasks doesn't exist on the internet, right? So the process data for how do you do disclosure schedules or what is market, right? Those are not things that are just kind of like on Reddit somewhere, right? And so what we do is we actually hire domain experts who sit down and say, these are the steps that I would take, right? And then you just chain the models on top of that, right? Or if there's a gap between what the model can do then you do fine-tuning right but the best way to do fine-tuning and post-training [12:11](https://realalpha.app/v/eXK-_yyQDMM/?t=731) is task-specific it's not here there was a large kind of ethos maybe in the legal industry and a bunch of other industries that somehow if you got all of the legal documents and then you just trained a model on that it would do law right um and and that's kind of Like saying, read all the, you know, your caseload and all of the textbooks in law school, and then just put you into like a profession, you somehow know how to do it. And that's not how it works. So much of it is like, you actually have to learn how to do the different steps, right? And then the other side of that is not only is the process difficult, but the evaluation is really difficult, right? [12:50](https://realalpha.app/v/eXK-_yyQDMM/?t=770) So you have to hire kind of like mid-level folks to do evaluation for a lot of these things because if the junior folks could do evaluation, they would be mid-level or they'd be senior, right? Like that's the reality. And a lot of what you do in a law firm or professional services is you're actually evaluating the work of junior folks, right? It's incredibly expensive. I'd argue that maybe 20% to 30% of the revenue of these places is doing that. [13:15](https://realalpha.app/v/eXK-_yyQDMM/?t=795) I want to go back to this concept of expanding and collapsing the surface, which I love.