Financial AI evals must predict real job performance

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 22:48

   Yeah, makes sense. Yeah, let's talk a little bit about evals. Evals is, yeah, something near and dear to my heart. I spend a lot of time working on evals. I would say that the prior that I would share with people is building a good eval is incredibly hard. That's not obvious to people who haven't worked on an eval before. To build an eval that actually correlates with the thing that you're actually trying to measure, like the skill that you're trying to measure, the capability you're trying to measure, very difficult.

2. 23:13

   Because whatever you put together is just a proxy for whatever that thing is, right? And it's just one dimension. So that's one. Two is it's very tedious to get the answer key or the way that you grade this correctly. And if you don't get the way that you grade the answers correctly, it's an eval that's very easy to hack, a very easy to reward hack. And therefore, you could score very highly on the eval, but actually do really poor on the actual underlying skill or objective that you're trying to measure, right?

3. 23:37

   And so the prior that I'll share is evals are really hard, not just in financial work, but just across the board. and what I would say is we don't think that there's a really good eval today for financial work. There's a bunch of existing prior work. I'll reference some of the work that we did which is a GDP eval which is a little bit more broadly in knowledge work. If you guys look at the Astra announcement we don't talk about GDP eval.

4. 24:01

   We know that the eval is completely saturated and the pieces that are not saturated are essentially broken. And so we actually don't report on that. We don't track it anymore. And so one of the tricky things for GPT-6 Astra was we didn't talk a whole lot about financial evals. We have some internally that we didn't talk too much about and didn't want to share at the time yet. They're coming. But there was no good way for us to show progress on financial tasks on these evals. There's a funny thing. I think there's this bias in evals where if the numbers look good, the researchers assume that the model's gotten better and they don't investigate it. If the model you've made a much more capable model and the eval numbers look bad or haven't increased in the scale that you want, then you go investigate.

5. 24:40

   So this is kind of like bias baked into investigation of these evals. Something like this happened for GBD6 where we ran GBD6 on a set of financial evals, and the numbers didn't really change much. We're like, oh, what's going on here? Let's take a look at this eval. We did a deep dive into these different evals, and lots of things were broken. Basically, we found that we had saturated the evals, and the questions that we were getting wrong were driven by incorrect answer keys, poorly specified prompts, or just some other flaw at the eval.

6. 25:05

   And so all that to say, one of the things that my team is working on currently is to try to build a much better eval that actually predicts the ability to automate the work of that specific profession. And so, yeah. And so the idea is like, we'll have something like an investment banking eval that predicts, you know, if you score well on this eval, you could probably automate a lot of the work of a junior analyst in investment banking.

7. 25:22

   We got to wrap here soon. I want to ask any call outs for people in the finance space, the latent space audience.