The alpha, twice a week
That was 2 min.
Get the best new moments in your inbox every Wednesday and Saturday. It's free.
Check your inbox
We sent a link to your address. Click it to finish signing up. Not there in a minute? Look in spam.
or
Unsupervised Learning · 2 min · Jun 4, 2025
Missing workplace context can be the main evaluation bottleneck
Brendan Foody
Brendan Foody says ranking candidates is easier when many do the same job, while unique roles bring confounding factors.
In this episode
Full analysis- 13:51AI labs are shifting from crowdsourcing to expert dataPro pick 3 min
- 18:55Mercor followed demand signals into AI data work4 min
- 35:00Foundation models may split between APIs and custom applications3 min
- 23:46Mercor's AI interviews increased onsite conversion2 min
- 26:44Knowledge workers may shift toward creating evals4 min
- 7:35AI can assess candidates while people focus on selling the role3 min
- 11:17Missing workplace context can be the main evaluation bottleneck2 min
- 5:13Mercor tunes candidate evaluations with customer outcomes3 min
“So we found that actually oftentimes just making sure the requisite data is in model context is a majority of the problem.”
Brendan Foodyat 12:37 · Unsupervised Learning
Takeaways
- Ranking is easier when many candidates do the same job.
- Unique roles bring more confounding factors into evaluations.
- Getting the relevant context into the model can solve much of the problem.
About this segment
Brendan Foody says ranking candidates is easier when many do the same job, while unique roles bring confounding factors. Often the bigger hurdle is collecting context people forget to supply.
Brendan Foody is co-founder and CEO of Mercor.
Co-founded Mercor, a labor marketplace supplying AI companies with expert data, paying tens of millions weekly.
- JS
Jordan Segall, Host · Partner, Redpoint
From “Mercor CEO: Evals Will Replace Knowledge Work, AI x Hiring Today & the Future of Data Labeling”, Jun 4, 2025. Watch the full episode
Thanks. We'll look at and fix what's off.
Transcript
Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.
- 11:23
them versus if you have a hundred people doing a very different job, right? Like founders, right? They're all working on something that's nuanced in one way or another. It's very difficult to pattern match what is the thing that they said or the thing that we learned that actually translated to the outcome, because there's just so many confounding variables in the equation. And so I think that it's going to be relatively easy for the larger pipelines of roles, like if you're hiring 20 account executives, stack ranking all of them, learning from those signals.
- 11:52
um and then the models are starting to be able to learn from uh these you know much more complicated things where everyone's working on something else uh like we're doing a ranking of a bunch of the the teal fellows and and that's that's like a fun case but it definitely is more challenging uh and relies more underlying reasoning capabilities of the models. Maybe just talk through, like, what are some of the challenges that emerge in doing that?
- 12:17
Yeah, well, it's basically that oftentimes there's a lot of things that aren't in model context. And so models struggle to learn from that. And people, like, forget to add it to model context. So maybe it's like, I heard my, like, friend said this good thing about using this company's product, right? Or these things that, you know, might not be making their way in, making sure that all, references are added, all of the interpersonal stuff that humans might pick up on. So we found that actually oftentimes just making sure the requisite data is in model context is a majority of the problem. Yeah. I guess in the future, maybe we're just recording every conversation with our smart glasses. Yeah. Easy enough to feed into the model. Bridgewater had it right all along. Yeah, exactly. Is that where we're headed? Is it just like going to be Bridgewater
- 13:03
at scale? We'll see. I mean, I think, of course, a lot of companies will be adverse to that. And I think there will be regulatory reasons and legal reasons people don't want to do that. But I also think there's just going to be better processes for how models help get this information context, right? Like maybe it's AI doing an exit interview of the manager and the people on the teams to help better understand what was going on, because all the people have so many details in their head around this that we just need to get into the models for them to be able to make these superhuman predictions.