The alpha, twice a week
That was 3 min.
Get the best new moments in your inbox every Wednesday and Saturday. It's free.
Check your inbox
We sent a link to your address. Click it to finish signing up. Not there in a minute? Look in spam.
or
TBPN · 3 min · Aug 7, 2025
GPT-5 generated four to ten times more tokens in one test
Claire Vo
Claire Vo says Chat PRD saw GPT-5 generate four to ten times as many tokens as earlier models.
In this episode
Full analysis- 1:09:03ChatGPT emerged after OpenAI tested whether anyone would pay6 min
- 1:44:43OpenAI prioritizes adoption and compute over near-term cash flow5 min
- 3:08:28Post-training improvements made GPT-5 more reliablePro pick 5 min
- 1:18:07Test-time compute compounds with pretraining and reinforcement learning4 min
- 1:27:06AI agents may help with personal tasks people must coordinate themselves3 min
- 2:19:52Charlie wrote 98% of merged code in one August test4 min
- 2:37:44Agents build context through repeated tool calls3 min
- 3:31:26GPT-5 generated four to ten times more tokens in one test3 min
“I'm seeing four to 10 x the number of tokens generated”
Claire Voat 3:32:13 · TBPN
Takeaways
- Chat PRD is A/B testing each model rollout with users.
- More tokens can mean overly long business writing.
- Vo says GPT-5 leans toward tactical execution over strategy.
About this segment
Claire Vo says Chat PRD saw GPT-5 generate four to ten times as many tokens as earlier models. She is testing whether the extra output helps business writing, where concise strategic answers may matter more.
Founder, ChatPRD
Built ChatPRD, an AI product copilot that helps teams write better docs and ship faster products.
John Coogan, Host · Co-host, TBPN
Co-founded Soylent and Lucy; OpenAI bought his show TBPN in April 2026.
Jordi Hays, Host · Co-host, TBPN
Founded Branded Native; co-founded fintech Capital, which Rho bought.
From “OpenAI LIVE: Greg Brockman, Mark Chen & More”, Aug 7, 2025. Watch the full episode
Thanks. We'll look at and fix what's off.
Transcript
From the transcript TBPN publishes with the episode.
- 3:31:34
How how much like how much time do you have to kind of move users over to five Yeah. So I I mean, I think we have tons of time from the API side to move Yeah. Move users. And in fact, you know, our strategy at Chat purity is not to just upgrade to the latest model. I know Zach at warp said like, why wouldn't you want the latest intelligence? And the reality is because we're doing a lot of business strategy and business writing, I actually wanna validate with our users that they're they're getting the quality of strategic thinking output writing that they really want. So we actually a b test every single model rollout and really evaluate for user quality, token generation, all those things. And, you know, looking early on, it yaps. Man, this thing just wants to go through tokens. Right now, I'm seeing four to 10 x the number of tokens generated between the, you know, four generation models and five.
- 3:32:29
Wow. And when you're in a business context, you do not always want longer Yeah. Words, you know? And so it'll be really interesting there. It is certainly focused on execution. So I, you know, I've heard a lot from the OpenAI team. It's steerable. Yes. And. Its natural inclination is to drive you towards like, how, what, very tactical, very specific. And so if you're trying to zoom back out at a, strategic level or focus on a business initiative, it's actually a little harder to tune in that direction. So, you I think there's a lot of positive things for me as somebody who uses AgenTic coding platforms, who writes a lot of code. It's my daily driver now. I love it. But for other use cases, I think it's gonna take some time to figure out if it really is optimal in use cases where intelligence actually isn't the differentiating capability.
- 3:33:21
Yeah. It's very interesting to think the the best the best product manager is not the one that writes the most the longest doc.
- 3:33:29
No. And you don't send your engineer into your executive meeting. Like, I and I I really am looking forward to the time where we're not getting these number based models where actually I can get like GPT developer or g t GPT strategist where they're pre tuned and trained for the role they're gonna play as opposed to general purpose, but clearly oriented towards a set of set of tasks. And I just think if you look at this model, it was oriented towards an engineer, software engineering, at least in my experience.
- 3:34:04
So have you been tempted to launch any type of agent, like agentic coding products? You are you guys are obviously response at Chet PRD responsible for creating documentation. And if you look at the other guests that have joined today, many of them are competing with each other in different ways and trying to own different parts of the stack. You guys have seemingly stayed really really laser focused and no one else is doing anything like you're doing at least on the show today. But talk about like picking your your lane and