# Measure AI spend by output, not token counts

Markdown copy of https://realalpha.app/s/measure-ai-spend-by-output-not-token-counts/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Scott Wu](https://realalpha.app/builders/scott-wu/) · Scott Wu is co-founder and CEO of Cognition. · Co-founded Cognition, the company behind Devin, an autonomous software engineer; three-time IOI gold medalist.
- Host: [David Senra](https://realalpha.app/people/david-senra/) · Host, Founders · Has read an entrepreneur's biography every week for Founders since 2016.
- Show: Founders
- Episode: [The Future of Software & AI | Cognition’s Scott Wu](https://realalpha.app/v/PYobZzjW_ic/), Jun 28, 2026
- Moment: 48:31 to 52:21 of the episode (4 min), playing on https://realalpha.app/s/measure-ai-spend-by-output-not-token-counts/; the episode from there: https://realalpha.app/v/PYobZzjW_ic/?t=2911

## Quote

> “if your engineers are actually able to **ship three times more**, you know, then it's very clearly worth it.”
> — [Scott Wu](https://realalpha.app/builders/scott-wu/) at [49:16](https://realalpha.app/v/PYobZzjW_ic/?t=2956) · Founders

## Takeaways

1. Token spend alone does not show whether engineers create more value.
2. Compare costs with shipped tickets, projects and internalized work.
3. Use stronger models for hard tasks and faster, cheaper ones for routine work.

## About this segment

Scott Wu says token use is a poor proxy for value: teams should compare AI costs with the engineering work shipped. Devin routes subtasks to models suited to their difficulty and speed.

From “The Future of Software & AI | Cognition’s Scott Wu”, Jun 28, 2026. [Watch the full episode](https://realalpha.app/v/PYobZzjW_ic/)

## Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[48:35](https://realalpha.app/v/PYobZzjW_ic/?t=2915) which I think are really important. One, obviously, is just like really working with them on the very clearly defining the ROI, which I think is a very important, I mean, it's like everyone's talking about this, right? Like over the last few months, like people are going crazy on their token budgets and, you know, one engineer can spend so much. And, you know, you want to know that that's actually doing real value for you and you want to be able to identify where you are. Do you think all this stuff is a little crazy? I think it is directionally correct, but I think there's some, you know, there are definitely some places where people have gotten carried away. You know, people talk about, oh, like, yeah, like, we rank our engineers by how many tokens they're spending. Well, let's try and rank people by how much output they're actually

[49:13](https://realalpha.app/v/PYobZzjW_ic/?t=2953) producing or how much good work, you know, is actually getting done, you know, but obviously I think that, like, the math works out in terms of, like, you know, the GPUs are expensive, but, like, if your engineers are actually able to ship three times more, you know, then it's very clearly worth it. You just want to make sure you're doing it the right way, right? Obviously a lot there on just like tying to specific outcomes, tying to, okay, what are the actual tickets that are getting done? What are the projects and the initiatives? And like this project, which was scooped out for 18 months and was going to be handed off to, you know, an outsourced contractor and was going to cost you 15 million. Like, let's just talk about how that you do this all internally with your own team and you do it for 1 million and you get it all done in three months, you know, stuff like

[49:52](https://realalpha.app/v/PYobZzjW_ic/?t=2992) that, which I think is super important. The second thing which I'll call out especially is just being neutral. A lot of discussion, obviously. What do you mean by that? Being neutral with respect to all the labs, the models themselves. Yeah, I think you said you like being Switzerland. Yeah, we like being Switzerland, exactly. And so I think it's like an important thing of, you know, we are just as incentivized as they are to figure out how to make their token spend efficient, right?

[50:17](https://realalpha.app/v/PYobZzjW_ic/?t=3017) And so Devin is purposely meant to be, you know, a compound model system. And so all of these different things, you know, for the right task or even the right subtask of a task or something like that. Describe what a compound model system is. Yeah, yeah. So for each different part of your use case, I mean, imagine, let's say you tell Devin, hey, customer just reported this bug, you know, there's a Jira ticket, can you take a look at the ticket and like go and solve the whole thing, right? What does Devin actually go do, right? First of all, Devin's probably going and like investigating, understanding like what does the report say, what's going on? Second of all, probably, you know, as any engineer should, like it's going to go and try to reproduce the bug itself, right?

[50:54](https://realalpha.app/v/PYobZzjW_ic/?t=3054) So it'll say, okay, let me spin up the product locally. We click around, try to follow the same steps that they followed and see if I can make the bug happen as well, right? Then if you did that, then like, okay, now you're looking at the logs. You're trying to figure out what went wrong. You're pinpointing what are the particular files or what was the flow of, you know, what was the command flow that led to this.

[51:12](https://realalpha.app/v/PYobZzjW_ic/?t=3072) Then you do the debugging. Then you go and test it again. Make sure, you know, all these steps, right? Then you put it up for review. And it turns out that there are different models that are good for different parts of these tasks, right? And even across different tasks, obviously, also, it's like some tasks are these really crazy hard ones where you want to actually use max thinking and you want to use the very best models you can get your hands on in the world, right?

[51:34](https://realalpha.app/v/PYobZzjW_ic/?t=3094) And then many other tasks are, you know, boilerplate enough or repetitive enough that what you care about is just getting it done really fast, getting an immediate answer, having the ability to verify that it was correct, but then beyond that, making sure that it was like as cheap and fast and efficient as possible, right? And so Devin, rather than being pegged to one model and saying, oh, we're only going to serve you GPT for this or we're only going to serve you Opfist for this, Devin can use any of the different models it has in its arsenal, which include all of these models from Anthropic, OpenAI, Google, et cetera, but also our own models, right, or open source models out there.

[52:08](https://realalpha.app/v/PYobZzjW_ic/?t=3128) And it will, you know, dynamically go and choose these models for these tasks. How do you think about this? Like, you're a customer of them, but also competitor with. Yeah.