The alpha, twice a week
That was 3 min.
Get the best new moments in your inbox every Wednesday and Saturday. It's free.
Check your inbox
We sent a link to your address. Click it to finish signing up. Not there in a minute? Look in spam.
or
Latent Space · 3 min · Oct 31, 2026
Moving up the analyst stack makes evaluation harder
Joseph Kim expects AI to move from analyst tasks toward associate and MD work as capabilities improve.
In this episode
Full episode- 4:10Bundled market data fixed connector failuresPro pick 3 min
- 7:00Most finance AI product work is buildable by startups2 min
- 22:45Financial AI evals must predict real job performance3 min
- 9:16Financial AI can fit into existing review hierarchies3 min
- 19:01Moving up the analyst stack makes evaluation harder3 min
- 15:45Pairwise human feedback can train models on subjective taste3 min
- 12:32Factual errors are more dangerous than weak visual judgment3 min
- 1:57Finance training improves the shared model, not just one product2 min
“We really want to get the P99.9 type of feedback to make the model as good as humans.”
Takeaways
- Current models suit clear instructions like building slides or models.
- Investment ideas can take years to validate.
- Progress depends on model scaling and expert feedback.
About this segment
Joseph Kim expects AI to move from analyst tasks toward associate and MD work as capabilities improve. Higher-level investing is harder to evaluate because outcomes are fuzzy and may take years to verify.
Joseph Kim is a leads ChatGPT for Financial Services at OpenAI.
Shawn Wang (swyx), Host · Co-founder, AI Engineer
Wrote “The Rise of the AI Engineer” and founded Smol AI, maker of AI News.
From “Building ChatGPT for Finance — Joseph Kim, OpenAI's Head of Financial Services”, Oct 31, 2026. Watch the full episode
Thanks. We'll look at and fix what's off.
Transcript
Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.
- 19:04
the models that you're training will become much better at investment thesis and sort of generating ideas for investment and trading. But obviously that's something that humans or experienced kind of financial professionals are very good at and they kind of have an intuition for that kind of thing. But it's often counter to the prevailing wisdom out there. Otherwise there's less of a chance of it making a great return.
- 19:29
So how are you tackling that problem in terms of training the agent to do that kind of work? Yeah. I would say that like up until date, the vast majority of applications of AI and finance have been really focused on, I would say like the junior analysts work. And I think that's for two reasons. One is you have to build the product based on the like capabilities of the model of today. And so if you look at the capabilities of the model today, they're really quite good at taking pretty concrete instructions of what to do and kind of executing, right? So build me a slide. I want it to look this way. You know, make it look good.
- 20:03
Use our template. These types of things. The model is quite capable of doing those things. And so therefore, we've built product experiences around that. And the other thing I would call out is there's a strong preference to make the model good at things that humans are not good at or don't like to do, right? And so, you know, most humans I talk to, there's obviously exceptions, but a lot of the humans I talk to, like, humans I talk to, people I talk to, you know, they really don't like building Excel models. It's a tedious kind of work and it's very detail-oriented. It's easy to get wrong and similarly with slides.
- 20:32
You're copying logos, you're placing them very precisely on a specific part of the page. It's very tedious work that a lot of people don't like to do. That's been part of the impetus for us to go and focus on some of these tasks. It's a combination of two of those things. The existing molecular is pretty good at certain types of tasks and also human preferences of what they like to do and not like to do helps drive the direction of the product as well. It's a combination of those two things.
- 20:53
I think that we've solved a lot of the low hanging fruit in those two things. And so going into 2027, we're going to start to move up the stack, right? We're going to go from the tasks that a junior analyst can do to some of the tasks that an associate can do to some of the tasks that an MD would do and move up that chain. And as you move up the chain, what I would say is the objectives become much fuzzier.
- 21:12
It goes from this is clearly a good or bad slide or something that would put in front of a client or not to, well, I don't know if this investment idea is good. And it would take me three years of waiting to really be able to give you a clear, concrete answer of what the market has done. And so the objectives start to become fuzzier. And so the way you go about training the model and the way you think about how to make the model better at those things becomes more difficult.
- 21:32
It becomes more difficult from a research perspective. And so, but it's not impossible is what I would say. And so what it takes is a lot of research focus. It takes a lot of wonderful data. It takes a lot of feedback from experts in the field. We really want to get the P99.9 type of feedback to make the model as good as humans. but I don't think it's an insurmountable problem. And what it takes is continued scaling of the base model and focus.
- 21:57
Focus on the research labs to think that this is an important problem to go solve. And one of the things that I'm viewing on my team is making this an area of focus and priority for OpenAI. And I think because of the combination of those two things, I feel confident we'll make a lot of strides in 2027. Do you have any comments on any of the current financial benchmarks?