Real usage and task costs are better signals than model branding

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 24:01

   This past week, OX Alpha went a bit viral, which was basically an anonymous model. No one knew who made it. Now we figured out that it's from Z.ai. What did, watching that experience, what did that prove to you about like model preferences and how much people care about brand and some of these things? yeah so when it was in stealth mode which is a program that we have that we started back in 2024 and why did you start that so in in uh what uh what we we worked with open ai to to do the stealth model program for the upcoming launch of gpt 4.1 and we wanted to provide, we wanted to basically let users evaluate a model without showing bias around the brand that made it and get a better, a way of providing interesting feedback to the model lab, but also to create a way for people to build their own benchmarks and see how the stealth model performs

2. 25:21

   on community benchmarks, benchmarks other than the things the model labs normally evaluate internally. So it was a good way of kind of doing revealed preferences without implicit bias that people have. And we've continued the program and done it with many model labs since then. And like this past month, we did it with Z.ai for their launch of GLM 5.3 Flash. The model was called Ox Alpha. and or yeah ox alpha and a lot of people tried it more unique users tried it than any frontier model um it was very very popular it was the largest model launch in open routers history in terms of tokens and it was just very capable um we also offer the stealth models for free and a lot of developers just found it like very capable at long horizon work and very good pros as well. And when it exited stealth mode, it's still pretty cheap compared to other models,

3. 26:30

   right? Yeah. GLM 5.3 flash is very, very good cost per task performance. And I believe they did about five trillion tokens in their first four days, which is very, very high after the model was revealed. A lot of the industry is very focused on cost per token, like price per token. We're guiding the industry towards thinking about cost per task and cost per session because the models have such different profiles around token efficiency and it, you know, it matters like how much it costs to complete the task, not price per token.

4. 27:15

   So if you look on our rankings page or on our model pages, you can now see like estimates of how much it actually costs to have like a reasonable session in cloud code or codex or any agent using each model. and if you go to that spot in our rankings page today you can see glm 5.3 flash like ranked really high wow um it was also just cool to see that like cultural conversation around it everyone was so excited and curious um and that was very interesting um if you were to look very far out do you think people will just like pay for an outcome like they like this will all be sort of like electricity you don't care where it's made or how it's made but you just are like oh i'm willing to spend this much on getting This outcome, and then you guys handle all of that, and so people don't care about brand, or frontier model, or Chinese open weight model, or these different ideas that are a big deal right now.

5. 28:10

   I think there will be markets for both people who care about the model and kind of intelligence that is being used behind the scenes, and people who don't. A lot of consumer apps, for example, people don't really care about which model they're interacting with under the hood. They just care about the job getting done. B2B is a little different. Businesses do tend to care about the model and actually, like, want to understand and build routers themselves.

6. 28:40

   Like, I believe most companies are going to become model routers of a new type. And, you know, we'll see, we've been saying this since, like, 2024. Like, a lot, like, all startups, all software startups will become model routers. They want to know which intelligence inputs they're using, and they want to understand how those choices affect their cost of goods sold, their COGS, and their key performance indicators, their KPIs that they're trying to maximize for the product.

7. 29:19

   And choosing the models and understanding how the models are prompted are the inputs to those equations. And so you want control as a business, but you don't really care as much as an end user sometimes.