# Charlie wrote 98% of merged code in one August test

Markdown copy of https://realalpha.app/s/charlie-wrote-98-of-merged-code-in-one-august-test/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Riley Tomasek](https://realalpha.app/builders/riley-tomasek/) · Founder, Charlie Labs · Built Charlie, an autonomous TypeScript engineer; previously cofounded Flight, acquired by Figma, and founded Dexa AI.
- Host: [John Coogan](https://realalpha.app/people/john-coogan/) · Co-host, TBPN · Co-founded Soylent and Lucy; OpenAI bought his show TBPN in April 2026.
- Host: [Jordi Hays](https://realalpha.app/people/jordi-hays/) · Co-host, TBPN · Founded Branded Native; co-founded fintech Capital, which Rho bought.
- Show: TBPN
- Episode: [OpenAI LIVE: Greg Brockman, Mark Chen & More](https://realalpha.app/v/gaImbWPGgtU/), Aug 7, 2025
- Moment: 2:19:52 to 2:24:10 of the episode (4 min), playing on https://realalpha.app/s/charlie-wrote-98-of-merged-code-in-one-august-test/; the episode from there: https://realalpha.app/v/gaImbWPGgtU/?t=8392

## Quote

> “**98% of the code** that got merged into our code base was written by Charlie.”
> — [Riley Tomasek](https://realalpha.app/builders/riley-tomasek/) at [2:22:42](https://realalpha.app/v/gaImbWPGgtU/?t=8562) · TBPN

## Takeaways

1. Their coding-agent evals focus on hard and very hard tasks.
2. They score pull requests for correctness, quality and testability.
3. Tomasek says engineers need to plan well as agents write more code.

## About this segment

Riley Tomasek describes hard coding-agent evaluations and says Charlie wrote 98% of the code merged into its own codebase in an August test. He says hiring will favor planning over writing code by hand.

From “OpenAI LIVE: Greg Brockman, Mark Chen & More”, Aug 7, 2025. [Watch the full episode](https://realalpha.app/v/gaImbWPGgtU/)

## Transcript

From the transcript TBPN publishes with the episode.

[2:20:01](https://realalpha.app/v/gaImbWPGgtU/?t=8401) They're finding everything. Yeah. That's funny. And then and then talk to me about, the shape of those of the actual problems in the eval. Are you are you doing are there some easy questions, some hard questions, some some some extremely hard questions? Like, are you formulating those? What's the shape of an individual task? Is it scored out of, like, a 100? How do you think about developing a good eval?

[2:20:26](https://realalpha.app/v/gaImbWPGgtU/?t=8426) A mix of hard to very hard. The easy ones are just a waste of money and time at this point, especially with five. Like, there's a bunch that it's just not gonna get wrong. Yeah. Yeah. And then we're mostly doing the PR ones look kind of like SuiteBench in the sense that we're taking an issue to start with. But instead of giving the issue, like, in a Docker container already, we trigger a comment on the issue that says, hey, Charlie. Go make a PR for this. Mhmm. And then Charlie does his thing, and then the PR comes up.

[2:20:55](https://realalpha.app/v/gaImbWPGgtU/?t=8455) And then we score that PR against a whole bunch of things like correctness to a known solution that's correct as well as code quality, testability, and some softer things like descriptions.

[2:21:07](https://realalpha.app/v/gaImbWPGgtU/?t=8467) Who are the biggest who are the biggest customers or use or or users for, like, a TypeScript focused coding agent?

[2:21:16](https://realalpha.app/v/gaImbWPGgtU/?t=8476) It's a wide range of mostly modern apps. Like, pretty much any web app these days is gonna be like a Next. Js type app. And then all the way into, like, back end. Like, Charlie himself is written in TypeScript. Sure. Sure. Makes sense. And there's very little

[2:21:29](https://realalpha.app/v/gaImbWPGgtU/?t=8489) front end. Anything else, Troy? What else you got? I just wanna say I love the name Charlie. It's one of my favorite agent names Yeah. That we've had on the show.

[2:21:39](https://realalpha.app/v/gaImbWPGgtU/?t=8499) Yes. It's right up there with Pig and what was the other one? Well, don't think that was an agent. That was an agent. But But yeah. It's a it's a good one. Yeah. Congrats on locking Yeah. It What what what about what about cost and and that side of the business? Is there is there any movement there or anything that you where where you require movement or you need movement to really unlock new capabilities in the business or new markets?

[2:22:08](https://realalpha.app/v/gaImbWPGgtU/?t=8528) Not really for us because we're operating as at a human level. We do value based pricing, so we charge per PR or per commit. Mhmm. And because that's comparing to such expensive actions that humans are doing, the challenge for us is more actually moving up to the promise than doing it cheap. Yeah. Yeah. Are you having

[2:22:29](https://realalpha.app/v/gaImbWPGgtU/?t=8549) But then but then doesn't the cost reduction announced today? Isn't that great for business?

[2:22:35](https://realalpha.app/v/gaImbWPGgtU/?t=8555) Yeah. I mean, it's good overall, but, like, that's our problem is not that the models are expensive. It's Sure. That they're I mean, they're getting really smart, but I'll always take more. That's great. Never enough. We like, for instance For token the August, we've been testing, and 98% of the code that got merged into our code base was written by Charlie. Wow. Not 30, not fifty, ninety eight percent. And that's coming through PRs. That's not like autocomplete, you know, an ID type thing.

[2:23:03](https://realalpha.app/v/gaImbWPGgtU/?t=8583) That's crazy. Wow. Yeah. What, yeah, what does that mean for, like, the future of of like, who are you hiring? I imagine that you're still, you know, a an engineering heavy organization that's just puppeteering and orgus and orchestrating agents. But where do you see, like, the future of software development as a career path going?

[2:23:23](https://realalpha.app/v/gaImbWPGgtU/?t=8603) Yeah. Are are are new CS grads cooked? I think if they get really good at using the AI, no. If they try and take an approach of getting really good at writing code by hand, for sure. Yeah. What we're mostly looking for hiring is people who are able to see things at a much higher level and plan further out. Because with tools like Charlie, you can write so much more code so quickly that it's, like, it's more important to see where you're going and take the right path than it is to be able to write it quickly.

[2:23:56](https://realalpha.app/v/gaImbWPGgtU/?t=8636) Very cool. Well, thank you so much for stopping by. Good luck with the rest of your day, and congrats on a on an upgrade to everything that you do. Yeah.