Open models now handle most token traffic on Vercel

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 52:12

   we haven't talked about one of the most talked about subjects yet and that is the war between the optical war between a lot of wars we didn't know this is a war podcast but here we are it is Well, I do a lot of interviews with defense companies. Now, here we are. But, okay, of the many tweets that you tweet that go viral, one of them was on, because you and Vercel sees everything that's going on in the world, on the infrastructure side, the difference between open weight and closed usage.

2. 52:46

   So what are you seeing there now? Yeah, so for context, Vercell operates one of the largest token distribution networks in the world. So in the internet that I described earlier, the internet of pages, we had content distribution networks, CDNs, and we would transmit pages. Now the really interesting new thing is tokens. It's a unit of intelligence. And you need networks to distribute these, to get your tokens, to power your agents, to send inputs and to get outputs back.

3. 53:19

   And so developers come to Vercel to get tokens, to power their agents, and we're basically one of the largest token suppliers in the world. We aggregate all of the model providers in all the open models in the world, or most of them. And so we get a lot of really good data that then we actually give back to the world under Creative Commons. So like anyone can use this anonymized volumetric data to understand where the future is headed, right?

4. 53:50

   And the thing that's astonishing is since the beginning of the year, you know, when we started out the Vercel AI Gateway, is what we call this network, I think it was like mostly like open AI, Anthropic and Google. That's where developers and enterprises were getting their AI. In fact, Google and Gemini had an amazing time where like most enterprises, they had found the perfect balance between price and performance with Gemini models.

5. 54:21

   I think they did a great job in that and they continue to be very strong there. And then we had the rise of coding models. I think maybe people don't know this, but a lot of the tokens in the world today are being consumed by developers. And you can think of almost like the software factories, the agents that these developers are commanding. This is one of the biggest sources of token demand in the world.

6. 54:48

   But what's happened is that in the beginning of the year, most of the traffic was going to proprietary models. So OpenAI, Gemini, XAI, et cetera. and now the situation is almost entirely flipped 80 of tokens in our network and this is the tweet that go viral that went viral is open models and there's a few reasons behind this open models used to be terrible horrible at coding i had personally never even though i'm the biggest open source person in the world i created some of the most popular open source software in the world. I will continue to fight for open. Our mission really in this AI era is what I call, I will enable you and I will enable the largest companies in the world to own your open intelligence stack. I want you to not be tied and anchored to a model, to not have control over

7. 55:39

   your data, et cetera. But I couldn't live up to that myself because I was like, well, the best coding model is not like I can't even use this open thing. But then some of this Chinese labs started putting out stuff that was extremely competitive with the frontier. The Kimi models, even I actually used Kimi yesterday again, and I was like, this model is so good. Of course, it's not like every single benchmark and level of execution. There's a lot of things that it cannot do, but enterprises realized there's a lot of stuff that we do, like 80% of our traffic, and maybe that's what the data is showing, 80% of what I do requires a certain level of intelligence.

8. 56:22

   I don't need to hire Albert Einstein to solve every problem, like answer every support ticket and spell check my stuff and classify my data. And so the world is literally opening up to choice and saying I'm going to use the best model for each task. For example, something that's really common with developers today is You do need Einstein sometimes to, or the Einstein level of intelligence, to create a plan.

9. 56:54

   I'm going to write a document of how I'm going to approach this problem, and I need as much intelligence as possible. But then I need execution. I need to translate that plan into action, into code, into instructions, into operations. And basically every open model is amazing at that task. And so what I think the data is clearly telling us is that the future is going to be a portfolio of options. Enterprises also love to have basically the negotiation power of saying like, look, I'm not anchoring or verticalizing myself to a specific lab.

10. 57:32

    I'm implementing tools and protocols that make the model more of a module, like an engine that you can swap out. and so if by tomorrow Anthropic is not doing so well in this particular dimension, I just change it out and I go to the Rosalia Gateway and with one line of code the other thing that's kind of nuts is you would think that replacing models would be this massive engineering effort and in reality it's one line of code or a lot of what the world is researching right now is can we even make the model choice automatic?

11. 58:09

    There are a lot of promising directions and examples of places where you can just don't even ask. And I think much of my career has been, much of my success has been, I try to think backwards from the user experience first. And I think this world where you have to first choose a model in order to do a task is not very user friendly. I would much rather have my colleagues say, okay, this is my query.

12. 58:38

    This is my prompt. And then we can decide where to route based on context, based on the intelligence needs. And sometimes I do think for very, very high-end engineers, they need to choose a specific. They know a model very well and they have to choose it. So it's a combination of broadening access to every model under the sun and then also letting some people pick.