TBPN · 4 min · Aug 7, 2025

Charlie wrote 98% of merged code in one August test

Riley Tomasek · Founder, Charlie Labs

Riley Tomasek describes hard coding-agent evaluations and says Charlie wrote 98% of the code merged into its own codebase in an August test.

In this episode

1:09:03 ChatGPT emerged after OpenAI tested whether anyone would pay6 min
1:44:43 OpenAI prioritizes adoption and compute over near-term cash flow5 min
3:08:28 Post-training improvements made GPT-5 more reliablePro pick 5 min
1:18:07 Test-time compute compounds with pretraining and reinforcement learning4 min
1:27:06 AI agents may help with personal tasks people must coordinate themselves3 min
2:19:52Charlie wrote 98% of merged code in one August test4 min
2:37:44 Agents build context through repeated tool calls3 min
3:31:26 GPT-5 generated four to ten times more tokens in one test3 min

“98% of the code that got merged into our code base was written by Charlie.”

Riley Tomasekat 2:22:42 · TBPN

Takeaways

Their coding-agent evals focus on hard and very hard tasks.
They score pull requests for correctness, quality and testability.
Tomasek says engineers need to plan well as agents write more code.

About this segment

Riley Tomasek describes hard coding-agent evaluations and says Charlie wrote 98% of the code merged into its own codebase in an August test. He says hiring will favor planning over writing code by hand.

Founder, Charlie Labs

Built Charlie, an autonomous TypeScript engineer; previously cofounded Flight, acquired by Figma, and founded Dexa AI.

John Coogan, Host · Co-host, TBPN
Co-founded Soylent and Lucy; OpenAI bought his show TBPN in April 2026.
Jordi Hays, Host · Co-host, TBPN
Founded Branded Native; co-founded fintech Capital, which Rho bought.

From “OpenAI LIVE: Greg Brockman, Mark Chen & More”, Aug 7, 2025. Watch the full episode

Answers “Which jobs will AI replace, and which are safe?” See all 132 answers

Answers “Will AI replace software engineers?” See all 40 answers

Answers “Are AI agents ready for real work?” See all 104 answers

Transcript

From the transcript TBPN publishes with the episode.

2:20:01
They're finding everything. Yeah. That's funny. And then and then talk to me about, the shape of those of the actual problems in the eval. Are you are you doing are there some easy questions, some hard questions, some some some extremely hard questions? Like, are you formulating those? What's the shape of an individual task? Is it scored out of, like, a 100? How do you think about developing a good eval?
2:20:26
A mix of hard to very hard. The easy ones are just a waste of money and time at this point, especially with five. Like, there's a bunch that it's just not gonna get wrong. Yeah. Yeah. And then we're mostly doing the PR ones look kind of like SuiteBench in the sense that we're taking an issue to start with. But instead of giving the issue, like, in a Docker container already, we trigger a comment on the issue that says, hey, Charlie. Go make a PR for this. Mhmm. And then Charlie does his thing, and then the PR comes up.
2:20:55
And then we score that PR against a whole bunch of things like correctness to a known solution that's correct as well as code quality, testability, and some softer things like descriptions.
2:21:07
Who are the biggest who are the biggest customers or use or or users for, like, a TypeScript focused coding agent?
2:21:16
It's a wide range of mostly modern apps. Like, pretty much any web app these days is gonna be like a Next. Js type app. And then all the way into, like, back end. Like, Charlie himself is written in TypeScript. Sure. Sure. Makes sense. And there's very little
2:21:29
front end. Anything else, Troy? What else you got? I just wanna say I love the name Charlie. It's one of my favorite agent names Yeah. That we've had on the show.
2:21:39
Yes. It's right up there with Pig and what was the other one? Well, don't think that was an agent. That was an agent. But But yeah. It's a it's a good one. Yeah. Congrats on locking Yeah. It What what what about what about cost and and that side of the business? Is there is there any movement there or anything that you where where you require movement or you need movement to really unlock new capabilities in the business or new markets?
2:22:08
Not really for us because we're operating as at a human level. We do value based pricing, so we charge per PR or per commit. Mhmm. And because that's comparing to such expensive actions that humans are doing, the challenge for us is more actually moving up to the promise than doing it cheap. Yeah. Yeah. Are you having
2:22:29
But then but then doesn't the cost reduction announced today? Isn't that great for business?
2:22:35
Yeah. I mean, it's good overall, but, like, that's our problem is not that the models are expensive. It's Sure. That they're I mean, they're getting really smart, but I'll always take more. That's great. Never enough. We like, for instance For token the August, we've been testing, and 98% of the code that got merged into our code base was written by Charlie. Wow. Not 30, not fifty, ninety eight percent. And that's coming through PRs. That's not like autocomplete, you know, an ID type thing.
2:23:03
That's crazy. Wow. Yeah. What, yeah, what does that mean for, like, the future of of like, who are you hiring? I imagine that you're still, you know, a an engineering heavy organization that's just puppeteering and orgus and orchestrating agents. But where do you see, like, the future of software development as a career path going?
2:23:23
Yeah. Are are are new CS grads cooked? I think if they get really good at using the AI, no. If they try and take an approach of getting really good at writing code by hand, for sure. Yeah. What we're mostly looking for hiring is people who are able to see things at a much higher level and plan further out. Because with tools like Charlie, you can write so much more code so quickly that it's, like, it's more important to see where you're going and take the right path than it is to be able to write it quickly.
2:23:56
Very cool. Well, thank you so much for stopping by. Good luck with the rest of your day, and congrats on a on an upgrade to everything that you do. Yeah.