Free with a sign-in.
The best 11 min of 44
Mercor CEO: Evals Will Replace Knowledge Work, AI x Hiring Today & the Future of Data Labeling
Brendan Foody argues that as AI takes on more tasks, humans will increasingly build evaluations to teach models what they cannot yet do.
Unsupervised Learning · Jacob Effron & Patrick Chase & Erica Brescia & Jordan Segall
- Brendan Foody is co-founder and CEO of Mercor.
The best minutes
- 13:51AI labs are shifting from crowdsourcing to expert dataPro pick 3 min
- 18:55Mercor followed demand signals into AI data work4 min
- 35:00Foundation models may split between APIs and custom applications3 min
Every moment plays free. PRO marks a write-up that's in Pro.
Also worth watching
- 23:46Mercor's AI interviews increased onsite conversion2 min
- 26:44Knowledge workers may shift toward creating evals4 min
- 7:35AI can assess candidates while people focus on selling the role3 min
- 11:17Missing workplace context can be the main evaluation bottleneck2 min
- 5:13Mercor tunes candidate evaluations with customer outcomes3 min
Skip the opening four-minute intro and broad AI-hiring overview, the founder's college and fundraising story, and the closing quickfire round.
Full analysis · 5 min read
Real Alpha · Nov 8
Brendan Foody expects human evals to grow by orders of magnitude
- Foody expects orders-of-magnitude growth in the human data and evaluations market.
- His rationale is shifting demand toward experts, but Mercor sells into that market.
- Teams should test evaluation quality against job outcomes before expanding budgets.
Free with an account
Read the rest of the analysis
6 sections, the 5 other voices it tests Foody against, and what it means for you. Every episode we cut gets one.
Check your inbox
We sent a link to your address. Click it to finish signing up. Not there in a minute? Look in spam.
or
Signed up before? The same buttons sign you in. The episode, its moments and the transcript stay free.
Models pushed labs toward expert data work
Foody’s central change is a move from plentiful generic labeling toward scarce expert judgment. That is narrower than saying AI makes human data obsolete.
When ChatGPT launched, he said, simpler models were easy to trip up. Large crowds could produce supervised examples and preference comparisons at volume.
But stronger models made routine annotations less useful, in his account. The next work asks specialists to expose failures researchers cannot easily see.
Ali Ansari, micro1’s CEO, said on Sabrina Halper Show in October 2026 that experts keep creating knowledge as AI automates work. That supports recurring data needs, but not Foody’s market-size forecast.
Foody expects human data and evaluations to increase by orders of magnitude over time. The episode gives no baseline, forecast method, or estimate of expert hours per model improvement.
“as the models got really good, that crowdsourcing model started to break”
Our read
The episode supports a shift from quantity toward expert contribution. It does not establish the scale or economics of future demand.
Domain experts help researchers define useful tests
In unfamiliar fields, researchers may not know what good work looks like. Foody says teams need experts when tasks exceed machine-learning researchers’ knowledge.
Take the doctor example: medical specialists help create assessments for what makes a good doctor. The expert’s role is not merely supplying answers to label.
They help choose meaningful tasks, recognize model mistakes, and turn those mistakes into training data. That puts evaluation design inside the work, not simply in an annotation queue.
Harrison Chase, LangChain’s CEO, said on 20VC in October 2026 that teams should start with context or a harness before custom models. His point reinforces the need to define the work before choosing the model.
Still, expert input can be expensive and difficult to standardize. The episode does not compare its cost with generic data or show when a custom evaluation pays for itself.
Our read
The opportunity is not just supplying specialists. It is translating their judgment into tests that reflect real work.
Performance loops fail when context stays missing
Mercor uses customer information about who performs well to improve candidate predictions. That gives evaluations a route toward outcomes, rather than scores that only measure agreement with an answer key.
But ranking many people doing the same job is easier than comparing people in different roles. Foody says unique work creates confounding factors that make performance harder to interpret.
He also says useful details often remain outside model context. People forget to add references, interpersonal information, and other signals they hold in memory.
Capturing more context is not free. Foody notes that companies may resist recording because of legal and regulatory concerns.
Joseph Kim, OpenAI’s financial-services lead, said on Latent Space in October 2026 that finance evaluations must predict job performance. He also said flawed prompts or answer keys can make tests misleading.
Our read
Outcome data is the right target, but missing context and unclear causality can break the feedback loop. A score matters only if it tracks the work the buyer cares about.
Mercor’s interview test shows promise, not proof
Foody says Mercor uses an AI interview for every role except executive roles. In one internal process, the company replaced a human case study with a standardized AI assessment.
He says conversion into onsite interviews then went up. That is a useful process signal, but not evidence by itself that hires performed better.
The episode gives no conversion rate, sample size, or comparison across employers. A funnel metric can improve while later job outcomes remain unknown.
Hiring also combines assessment with selling the role. Foody expects models to take more assessment work, leaving recruiters and managers more time with candidates they want to hire.
Reid Hoffman said on Possible in October 2026 that deeper AI interviews could help candidates overlooked by degree filters. That supports an access possibility, not proof of fairer selection or better matches.
Our read
Mercor’s experience supports testing standardized assessments against current screens. It leaves open whether scores predict durable performance or broaden access.
Foody expects workers to create more evaluations
Foody expects people to create evaluations for tasks models cannot yet do, rather than repeatedly perform those tasks themselves. He says maybe a majority of knowledge work is open-ended and hard to verify.
That could preserve a role for expertise as models improve. It does not establish that evaluation work will employ as many people as the tasks automation replaces.
Foody recommends learning quickly, working with AI, and focusing on areas where demand can expand. For example, he thinks demand could support building 100 or 1,000 times more software.
He does not mean 100 or 1,000 times as many web apps. He points instead to more feature iteration and better algorithms, while saying accounting demand is probably more fixed.
Mark Cuban described on The Diary Of A CEO in October 2026 how agents handle research and follow-up work he once could not get to. He argued that added capacity can help people build new things and hire for them.
Our read
Productivity can create work where demand expands, but that is not a general employment forecast. Foody’s software example remains a possibility, not measured evidence.
Custom applications may capture value beyond APIs
Foody expects general model APIs to become more commoditized. He thinks one of the leading labs is likely to lead that market, while value shifts toward applications built around customer context.
That outlook fits Mercor’s position. The company supplies expert labor and evaluation infrastructure, so it has a direct commercial interest in a larger market for custom evaluations.
The episode introduction says Mercor recently raised $100 million. That funding detail establishes company scale, but it does not validate Foody’s forecast.
For example, Foody imagines trading firms testing whether specific analyses lead to successful trades. Joseph Kim, OpenAI’s financial-services lead, said on Latent Space in October 2026 that higher-level investing is harder to evaluate because outcomes are fuzzy and may take years to verify.
That is the hard test for custom applications. The episode does not show whether customers recover evaluation costs through better decisions or more capable models.
Our read
Customer-specific evaluations could differentiate applications, but only if they track outcomes buyers value. In finance, proving that link may take years.
What it means for you
- Track evaluation yield. For each expert-data project, record how often an evaluation exposes a failure that changes training or product decisions. Compare that with the volume of work purchased.
- Audit outcome labels. Check which performance signals reach the model and which stay in managers’ heads. Start with roles where several employees do comparable work.
- Compare hiring screens. Compare standardized assessments with existing screens using onsite conversion and later job performance. Keep the sample size and role scope visible.
- Separate data budgets. Track spending on bulk examples separately from expert-designed evaluations. Set evidence requirements before expanding either budget.
What to watch
- Whether labs shift spending from high-volume labeling toward expert-designed evaluations.
- Whether Mercor’s standardized interviews predict job performance beyond onsite conversion.
- Whether human evaluation work grows as models absorb tasks, or mainly changes who performs that work.
Send this to the product or operations lead budgeting for AI evaluation.
Watch the three minutes where Foody explains why crowdsourced labeling stopped helping.
Written by Real Alpha from the episode's transcript and the other moments it links, then checked against the transcript. Quotes are the speaker's exact words.
Minute by minute
A Pro write-up, open to you here. Try Pro free for 7 days
A Pro write-up. The minute by minute is in Real Alpha Pro; the episode, its moments and the transcript stay free.
Read it with Pro, 7 days free- 1:20Models can evaluate text well
Foody says models are close to superhuman at evaluating information expressed in text, including interview transcripts, written assessments, and résumés. He recalls an early AI interviewer prototype that hallucinated every two or three questions; reasoning models later improved at handling context. He says models still struggle with human judgments such as whether someone seems genuine or enjoyable to work with.
- 6:29Hiring needs richer candidate context
Foody says AI can notice résumé details that human reviewers miss, such as personal projects or experience in the country where a candidate would work. He expects models to handle more of the assessment while recruiters focus on selling candidates on roles; dynamic, in-depth interviews can also make assessments harder to game. He says explainability builds trust and that adding missing references and interpersonal details to model context often solves much of the problem.
- 13:40Expert labeling is replacing crowdsourcing
Foody says early model improvement relied on high-volume crowdsourced SFT and preference feedback, but as models improved, labs needed highly qualified experts to create harder, real-world evaluations with researchers. He expects demand for human data and evaluations to grow by orders of magnitude because open-ended fields, such as judging founders, are harder to verify than math or code.
- 18:44A marketplace wedge can expand labor access
Foody says Mercor is using demand for AI training work to build marketplace network effects, then plans to expand from data contracts into high-volume contractor hiring and full-time recruiting. He says an introduction to xAI in 2023 showed the team how urgently labs needed its engineers, before the broader market for human data had emerged. He argues that founders should follow strong customer demand rather than force product-market fit.
- 23:46AI interviews can standardize hiring
Foody says Mercor uses its AI interview process for every role except executive positions; for strategic project leads, replacing a human case study with a standardized AI interview increased onsite conversion. He says marketplace experts work with researchers to develop evaluations, and that video could help models detect signals such as interest or cheating. He expects more knowledge workers to create evaluations for tasks models cannot yet do.
- 28:13Fast learning matters more than fixed skills
Foody advises people to learn quickly and work with AI enough to understand where it succeeds and fails. He argues people should focus on fields with elastic demand: greater software productivity could lead companies to build more software and hire more engineers. He says Mercor chose end-to-end hiring over a copilot because the full process can learn from performance feedback.
- 32:05Mercor grew from international hiring
Foody says he and his co-founders met at 14 on a debate team, began hiring engineers in India for projects, and bootstrapped Mercor to a $1 million revenue run rate with $80,000 in profit after paying themselves. He says he left college to build the company, while his co-founders needed a seed round to satisfy their parents’ condition for dropping out. He says the team aimed to limit dilution to 5% in later rounds and invest in marketplace growth and model improvement.
- 35:00Model value will move to applications
Foody expects general-purpose model APIs to commoditize and sees more durable value in products that integrate customer context and serve specific applications. He says customized models could help trading firms evaluate whether analyses lead to successful trades, though he thinks those firms have been less advanced in post-training than expected. On policy, he argues regulators should prepare for models outperforming people at their jobs, which he says could become a major concern within two or three years.
- 40:08Evals remain underhyped
Foody calls evaluations underhyped and says some companies spending billions on legacy SFT and RLHF data may need to spend an order of magnitude less. He now expects AI software engineers that can produce successful pull requests sooner than he once thought, but argues that greater productivity could expand demand for software. If starting over, he says he would build custom agents for a knowledge-work vertical, likely in finance, chosen for positive impact.
Published Jun 4, 2025.
Transcript · 40 min read
Unsupervised Learning, hosted by Jacob Effron, Patrick Chase, Erica Brescia and Jordan Segall, with Brendan Foody.
Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.
Brandon Foodie is the co-founder and CEO of Mercore, a company building the infrastructure for AI-native labor markets. Mercore's platform is already being used to label data, screen talent, predict performance, and evaluate both human and AI candidates. It's a really interesting company at the intersection of recruiting, evals, and core to improving foundation models. Brandon's team recently raised $100 million, and they're working with some of the most sophisticated companies in AI today. Our conversation today hit a lot of interesting things, including what role humans will play in labor in the future. We talked about the types of data labeling that really matter to improve models going forward. Brendan reflected on Mercore's rapid ascent and some of the key decisions he made. And we also hit on where AI does and doesn't work in the hiring process today.
All in all, a really interesting conversation. I think you'll really enjoy it. Before we get to the episode, I just have one plug. So many of you have rated the show on Spotify or Apple, and thank you so much for that. If you're enjoying this, please consider leaving a rating on the show on either platform. Ratings help us grow, which help us to continue bringing on the best guests and keep the bar really high for the conversations. Now, here's Brendan Foody. Well, thanks so much for coming on the podcast. Really appreciate it. Yeah, thank you so much for having me on. I'm a big fan, so excited to chat.
Yeah, excited to have you here. I figured we'd start at like the highest level place, which is for our listeners, I'd love if you could just contextualize, like, where are we today? What's the state of AI evaluating talent? Like what works, what doesn't, what's going on? I'm amazed at how good it is. I think that everything that a human is able to evaluate over text, the models are close to superhuman at. Whether it be the transcripts of someone's interview, the assessments that they're filling out in a written way, or even the signals on their resume. And it's a fascinating dichotomy because so little of that has actually been distributed in the economy, right?
Read the whole transcript
And so there's just this huge green field associated with doing that. And it's one of the things we're really excited about working on and building out. Yeah. Were there things that didn't work pre-reasoning models that, like, you know, maybe talk about the last six months. Like, as these models have gotten better, what's, like, finally started to work for you guys? Yeah. I remember back in end of March of 2020 or 2023 when GPT-4 came out and we were, like, build our first prototype of AI interviewer and nothing worked, right? It was, like, the model would hallucinate every two or three questions and all of that.
And so it's just been riding this incredible tailwind over time. And I think the reasoning models are just, obviously, the knowledge in the models improved a lot in sort of the first year. And then the reasoning models have made them much better at particularly handling a lot of context, figuring out what matters, what to focus on, et cetera. It's been really cool. Still, there's multimodal things that the models aren't as good at just because it historically hasn't been as much of a focus of the labs. And it's a lot harder to do RL with. But we're excited about that being added soon. Yeah. What are the milestones that, like, Matt, you're like, I can't wait till the model can do X or Y?
Yeah, there's a handful of things. Like, there are certain things that humans are very good at. Like, this, like, vibe check of whether I would enjoy working with this person, whether this person is passionate and, like, they seem really genuine about what they're saying. that the model, it's really hard, right? It's hard for the best humans, let alone models. And so I'm really excited about that and building evals out for a good chunk of it. But whenever I read through like the reasoning chains of the models, I'm like in trying to decipher things that are eval, I'm always thinking like, wow, the model seems a lot more reasonable than like whatever researcher on our team was creating the eval, right?
And so it's really incredible how fast they've improved. And I think everyone obviously is seeing everything working in code, but we're just in the early innings of a lot of other domains that are taking off in an incredible way. And obviously it seems like a big part of what you're doing is basically coming up with evals for humans and how good they'll be at jobs. Obviously we have all the people creating like AI employees now. It's like, hey, agents are going to do this. You'll have an AI agent doing. You know, this set of tasks that an employee would do, do you guys play into this at all? Absolutely. So, I mean, we do a huge chunk of this. Maybe giving a little bit of the backstory of the company, the reason we started is that we felt like there were incredibly talented people all around the world that weren't getting opportunities. And the primary reason is that labor markets are very fragmented and that a candidate somewhere else in the world, maybe it's remotely in the U.S. or another country, was only applying to a handful of jobs. The company in San Francisco is considering a fraction of a percent of people because there's this like matching problem that they're solving manually. And through applying LLMs, we could solve this
matching problem so that we could build this global unified labor market that every candidate applies to and every company hires from. But then we realized that there is this huge takeoff in hiring people associated with these like new knowledge work roles and evaluating LLMs. And so now we hire all sorts of experts for the top AI labs that use our technology to help facilitate that, both for creating evals to evaluate our experts as well as to evaluate the models and all of these agents that you're discussing.
“learning from all of the data we get from our customers of who's doing well, for what reasons”
“And I think about it as enabling human recruiters and hiring managers to spend all of their time on the candidates they want to hire”
Yeah, it's like an interim trust milestone on the way there. Exactly. No, it makes a ton of sense. And then obviously, today in kind of the first or one of the areas where you have a lot of fit on the data labeling side, there's kind of these clean feedback loops of like, you know, I imagine you could even score like how accurately and you probably have multiple people looking at the same pieces of data. Talk about some of the challenges maybe in translating this to like maybe more vague domains of human work. Totally. I mean, like venture capital. Yeah. Wait 15 years and then you get your feedback loop. Yeah. One way I think about it is like if you have a hundred people that are all doing the same job, it's very easy to stack rank
“So we found that actually oftentimes just making sure the requisite data is in model context is a majority of the problem.”
Yeah, there are certainly more and more both founders and all kinds of people that are bringing AI to their meetings. And so I think a lot of those meetings and interactions will be recorded for AI to learn from. Totally. I think that'll be interesting. We need you to take our transcripts and stack rank us against each other. Only if I come up on that. What do you think of the data labeling landscape today? How do you see the different players
“Because you needed really high quality people that would work directly with researchers to help them understand why is this model doing well?”
clearly I think one of the initial arbitrages and what inspired the company is you have these great coders, and they're all around the world, and they're not getting access to some of these jobs, and obviously that ended up being really important for coding data. Obviously, you've expanded into other areas as well. Coding, again, it's the perfect RL use case. It's probably also really perfect for evaluation. What have you had to change or improve as you've gone into some of these fuzzier domains and recruiting people in those areas? Yeah, I think that leaning on a lot heuristics of what a human would do manually is probably a good way to do it.
So, for example, if you want to automate being a consultant, how do you assess consultants that can help to do that, giving them a case study? Maybe that's specific to their background. Maybe it's like... You guys are all probably great coders, and so I imagine you know how to evaluate coders. If you're starting to get a doctor on the platform, how do you even know what the heuristics are for humans? Well, I think the point you're getting at is really interesting, which is that as you start to get into domains beyond the machine learning teams capabilities, they need to have these experts. We need to have doctors that are helping us create our doctor assessments and our evals for what makes a good doctor, as well as a bunch of other domains. And similarly, it's what the researchers need to do with all of their technology. When we were all going on LIMSYS, it was super easy to look at the high school level physics and say what problem was right or which one was slightly better. But when it's PhD level chemistry and the researcher doesn't have a PhD in chemistry, it's really hard to understand what's going on to interpret these evals to figure out how we can improve them. And so I think that that's the
other big shift to your question earlier around evaluations is that both for assessing our talent as well as the way the researchers assess models, it's just going to be this much more collaborative process and working with people to help trip up the model and improve capabilities. I've heard you talk before about how this kind of like short-term data label contract work is like, Like, it's kind of the perfect initial market for what you've done, and there's a massive amount of demand, and it's kind of this wedge to, like, eventually doing just kind of, like, end-to-end labor markets.
“you need to just look for the signs of the market, where it's like, wow, there's like gold to be found”
I guess you've expanded beyond coding, like, maybe to go back to the doctor example. Because I was struck by when you were describing it, one, you know, in some senses, like, evaling what a good doctor is, is, like, actually what you're eventually going to bring these people to the model companies. They're going to figure out, like, you know, is this the reasoning process that a good doctor would use?
What are you actually doing when you're working with someone to do evals? Yeah, I think that one of the key things that humans are better, a lot better at right now is like learning over time from the instructions, from the training, from all the feedback. And so, excuse me, looking for these proxies that people have demonstrated of like they're, you know, asking the right questions about the problem. They're going about thinking about it in the right way. They have signals in their background that indicate they've been in these high-performing environments where people are obviously learning significantly over time.
“And then we switched over to fully an AI process before the onsite and the conversion went up on the onsites.”
Obviously, you talked a little bit about using multimodal capabilities to determine passion and other things. Like, what other things are you thinking about with, like, you know, incorporating, like, video and, you know, other things that are futuristic for the platform? Yeah, one thing that I think about a lot is what role RL will have in the sort of, like, timeline to improve video capabilities in that RL is really good at these search problems.
Yeah. And video is just, like, a huge amount of tokens. That's why, like, models struggle with it. And so it's, like, in many ways, the search problem of, like, how do we look for the signal where that person was really excited about the particular thing? or if they cheated on the interview or like what other things we could find in multimodal context. And so I think a lot about how we can effectively, you know, create the right data to get the model to pay attention to those, as well as a lot of what the frontier labs are doing to improve those base capabilities. I mean, obviously it seems like, you know, even in the course of a few years, the end labeling market changed so much.
“I actually think it's highly probable that a huge chunk of knowledge work just trends towards creating evals.”
like, you know, there's probably a temptation that you could have built like a recruiter co-pilot or like partner, like built software for staffing agencies. Like you've obviously decided to go end to end. Like, was that obvious from the start? Like, how did that kind of come about? I think part of the start was just shaped by, I think we had a lot of benefit of just approaching the problem from first principles because we hadn't seen how it was done.
Like we knew the problem our friends wanted to solve is they wanted to work with a software engineer. And so we would just like handle everything associated with getting the software engineer that will perform well to be working with them. But in hindsight, I think that there's just many more businesses that will trend towards that because it doesn't make sense to build a co-pilot for a job that probably won't exist, at least in nearly the same way that it does. It probably makes more sense to have this end-to-end process automated in a way that it's able to learn from the feedback loops and make better predictions. Yeah. Now, obviously in your case, I think you benefited from like this daily market is actually perfect for, at a time of relatively nascent capabilities, you can do it kind of end-to-end, right? And I'm sure if that didn't exist, I imagine you might've had to go co-pilot for some of these other complex roles. I think this is absolutely right. Right. Cause it's like, if you're hiring full-time employees, then obviously, definitially people want to have them on their payroll. And so I think that is one thing that we were fortunate about is that our operating model and the way that we'd structured a lot of the business was very conducive to.
what the demand and the shift we were seeing in the market was. Initially, it sounds like you were helping find contractors for your friends. I assume at some point you were like, this is a side project, and then at some point it became the main thing. At what point was it like, yeah, I'm actually going to build this business for the next 20 years versus like, this is a cool thing I'm doing at the start of college? Well, the background is that I was always building companies in high school. I had a company that was doing pretty well, so I didn't want to go to college. I told my parents, no, I'm not going to go to college, and they did not like to hear that.
And so then eventually I appeased them. I applied to college, went to school, but I told them, like, I'm going to drop out. And they didn't really believe me. They figured that it was a safe bet once I'd agreed to go to school. And then I went to school and every semester I told them. They, like, blocked the term teal fellow on the computer. Like, please don't look this up. Yeah, every semester, you know, I tell them the same thing. And then eventually I dropped out without really giving them a heads up or telling them. Because I was like, I've been telling them for the last two years, right? You gave them a heads up.
I gave them a heads up. The signs were there. A long heads up, right? And so I think for me, it was that I knew that I just wanted to build a company. I was passionate about building things that have impact in the world rather than sitting through classes that didn't feel very productive. And I was, in many ways, just finding the right thing to spend my time on. I think with my co-founders, it was starting as a side project, wanting to make sure they had the evidence to justify their parents, their decision to drop out. And it's funny, part of their condition for dropping out was that we would raise money.
And even though we had this business that was doing a million-dollar revenue run rate, we'd profit at 80 after paying ourselves. It was making a lot of progress. That wasn't sufficient. The key was that we needed to raise our seed round. That's what keeps us VCs in business, parents wanting some validation. It's the credibility step. Well, that's a good segue. You recently raised a lot of money at $100 million round. Congrats. Thank you. What does that kind of allow you to do now? Or how did you think about, you know, when was the right time to go raise more capital? I'm sure people want to throw money at you all the time.
So, like, how do you think about when to cut off the spigot? Well, it's also interesting. The only time we went to raise money was really our seed round where we were like, okay, we need to raise money to justify dropping out. And then our Series A and our Series B. Exactly. Our Series A and our Series B were both preemptive. And so our thought process was that we wanted to keep dilution relatively low at 5% and sort of build up like a war chest so that we can invest in the product capabilities that we were talking about of like, how do we, you know, have referral incentives and all sorts of these creative consumer products that can build up the supply set of our marketplace, as well as investing in more post-training data to improve our models, performance prediction capabilities.
And in many ways, one of the largest blockers on our ML team is just creating more evals and more RL environments to improve our models, which happens to be very conducive to our business.
“I definitely am in the school of thought that OpenAI is and will continue to be a product company, not an API company.”
What's like the biggest unknown question you have like an AI right now that you feel like has you're like, God, if I knew the answer to this, it would make big implications for how I'm running the business today. I think it's what you said earlier of what humans will be doing in like five or 10 years. That's such a hard question to answer. And I think about that as the mission of the company in many ways.
And we have all sorts of intuitions, but the world is changing very fast. I think so many jobs are going to get automated that getting a better understanding of that and how we can help define humans' new opportunities and the role that they play in the economy is one of the most important things. Yeah. Is there more stuff that we should be doing from a policy perspective around this? How do you think about the role other institutions and society should play here? Absolutely. I think that so many regulators have been very focused on things that actually aren't as close to impacting American lives and that they're focused on competition with China, which, sure, it matters, but it's a lot less close to people's day to day.
They're focused on safety risks, which matter, but are a lot less close to people's day to day. I think the thing that everyone's going to start freaking out about in the next two or three years is just that there's these models that are significantly better than them at their jobs, and we need to figure out how they're going to fit into the economy. And that's something we know will happen. Right. It's not just this like low probability, high impact risk. And so I think that regulators need to be much more proactive around how we can plan for that future, around how we can stay like expectation management for the general public and what the world will look like in a few years.
Yeah, I guess it's just hard not knowing what we're retraining people for. Yeah, it is. Exactly. But I wish that there was a lot more conversation around that, right? And a lot more focus on what that next generation of jobs is going to look like and what guidance we should be giving to everyone as they're going through school and entering the workforce. Yeah. Well, we always like to end our interviews with a quickfire round where we get your You know, quick take on some overly broad questions that we'll stuff in at the end. And so maybe to start, we'd love, you know, what's one thing that's overhyped and one thing that's underhyped in the AI world today?
Oh, good question. I think that evals are underhyped very significantly. Even though they're hyped, I think they're still underhyped very significantly. One of the last bastions of human capabilities. Yeah, I think the one thing that's really overhyped is like SFT to RHF data or that like bucket of legacy data there's companies that are literally spending billions of dollars on it that don't need to be spending or need to be spending an order of magnitude less and that'll change what's one thing you've changed your mind on in the AI world last year interesting I think my timelines for automating software engineering have gone up significantly like I used to I used to be a little bit skeptical of hearing from researchers what their timelines are to having a really good AI software engineer that's able to write a PR that has a higher hit rate than a human.
And I think now it seems clear that that's coming later this year, sometimes in the first half of next year. And that's going to be really, really cool. Yeah. Do you think, I mean, obviously, like it seems like with some of these AI improvements, you know, it's like if you talked about them, what they were two years ago, you would have said, oh my God, it's going to change the world. And then they happened and it's like, okay, like that kind of adjusted things, but not like, do you feel like that's this like, a wow moment where like, you know, there's just mass change in employment on the software engineering side? Or is it one of those things that will feel like some 10% change or 20% change?
Well, I think the thing that frames it is the elasticity that we were talking about of the role and that I'm less worried about the short time horizon of engineering jobs, because I think giving them tools to make them more productive will just mean we build more software, but it'll definitely change the nature of the role and that people that are product minded, people that understand how to do the things that models might not be as good at or have more of a comparative advantage in the market. What AI startup are you most excited about besides Mercore? I'm really excited about OpenAI's coding capabilities, even though that's not a contrarian answer. I also think that there's going to be an immense amount of custom agents.
And so there's a company I'm friends with that's sort of like in stealth that I'm super excited about. All right. Well, you definitely can't share it on this podcast. When we stop recording, We'll harass you for what that is. Obviously, you're running a hugely impactful company. Let's say you were getting started today. You were just beginning and building some AI app. Totally different category. What else would you think would be fun to build right now? Or what else would you go spend time on? I think that I would choose a certain knowledge work vertical, probably something in finance that can be automated and build custom agents.
in that vertical to do so. You could build this AI trading firm. Yeah, I would probably try to choose something that I think is more positively impactful because I think that making sure that we get to the right valuation by the morning instead of the afternoon probably doesn't move the needle in the world. But yeah, I would choose something that I feel is super impactful to automate certain capabilities. But yeah, it's a cool world. Yeah. Well, I always want to leave the last word to you. It's been a fascinating conversation. Where can folks go to learn more about you, the work you're doing at Mercore?
The mic is yours. Anywhere you want to point to our listeners. Yeah, absolutely. Go to our website, mercore.com. We're hiring huge volumes of people for ourselves that are our customers, or smaller volumes for ourselves, huge volumes for our customers, and have all sorts of great opportunities that we would love to work with people on. Awesome. Well, thanks so much. That was super fun. Yeah, thank you so much. That was a lot of fun. We'll see you next time.
