Most enterprise workloads do not need frontier models

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 27:11

   probably will continue to it's hard to see why it wouldn't yeah how is your token spend to the degree that you want to share the details how has it evolved over recent months and how do you think about it i often ask people what's your ratio of token spend to payroll as one way of kind of getting a handle on it i'll say you know maybe one thing i'll just say is kind of peak for us It was kind of June, July.

2. 27:38

   It's been telling off. And part of that has been like learning a lot about – and frankly focusing on it. Like last year at this time, our sole focus was like, hey, shift everything to AI. Use models for everything. And that created a huge amount, a bunch of learning first. Like we learned a ton. Second, a huge increase in velocity in terms of just capability delivery. So think software engineering, a big part of what we do, shipping software products.

3. 28:12

   I think we shipped four major features in Q4 last year, 19 in Q1 and 26 in Q2 this year. So we saw a huge surge in our – and this is not minor things. These are major new capabilities that was a dramatic acceleration due to that kind of investment. We saw it in the token burn, but worth it for the impact. So that was great. But one of the things that we've learned is actually most of those things that we were doing, we could do just as well, a lot cheaper, with a better harness.

4. 28:43

   We built one for our own engineering organization that is, of course, full of all of our own context and is optimized for us. We built a gateway, right, a LLM router, so we're like, not all those jobs need to go to, at the time, whatever, Opus 4A or something, and we could reroute to lower-cost models for specific jobs. So we've actually seen – we're actually burning less today than we'd forecast in Q3 because of optimizations largely that we've been able to do.

5. 29:17

   So I think the reality for most organizations today is that they don't need frontier models for their enterprise workloads, like almost none of enterprise workloads. And I think about the real operations of a business as opposed to necessarily the creation of a new software asset. And those workloads can use models that are three years old for most of the stuff. Like, when you look, it's really boring what most of the enterprise workloads where AI can make a real advance happen today.

6. 29:49

   And a lot of that stuff can be done with deterministic code cheaper than it can be with a model at all. Or two, can be done with a lower-cost model, either because you're using open weight or you're using just an earlier version of one of the foundation models. and that's, I think, a key learning for us. We're seeing that in our own work. We're seeing that certainly with customers and this is why it's important to build your workflows, to build your agentic systems on a platform that makes it super easy for you to swap models to get the best of what's out there.

7. 30:27

   Maybe because capability improves, maybe just because you can do it cheaper with another model. And I think we're going to see without question that most enterprise workloads are operating at a much lower per token cost than, you know, what the frontier of the frontier is offering today. And at the frontier, people are cutting prices, right? Like, I think that's the other thing that's happening here. So there's going to be a lot of shift. Like, I don't think we were on a sustainable trajectory in the first half of this year, right? Like everybody, every CFO got the token bill in February and March. And it was like, obviously a no shit moment for the whole world. And so everybody started doing controls, whatever they were, capping token spin on a per person basis.

8. 31:15

   putting in a router that would allow you to route jobs to lower cost things, pulling back entirely from, you know, an AI first for everything everywhere strategy. We saw that in some companies. We saw big announcements about that. I think Microsoft, Meta, Uber, like all these companies who were like, wow, maybe went too far too fast. But I think you can get back to the point where you're really leveraging AI and everything.

9. 31:39

   You just need to do it in a way that is smart, right? And that's why you need platforms to help you do that. And that's the kind of job our systems is doing for our customers.