Open-weight models could handle most enterprise AI at lower cost

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 9:05

   happening currently. I would love to get both of your takes on the following concept. There's open source, there's closed source, there's frontier, there's pacing the frontier, there's Now, effectively, very reasonable alternatives that are open-weight at a minimum, your philosophy around the entire spectrum, and just give us a sense of how we should think about models and capabilities over the next couple of years.

2. 9:34

   Nikesh, you want to start? Sure. Look, I think earlier in the day, somebody said, don't restrict your teams in terms of capability in the experimentation phase. Don't start trying to cost optimize. Don't start trying to stitch your own in that phase because the teams who work for us 20,000 people have just discovered AI in the last 12 to 18 months. And it's not fully diffused across the organization.

3. 9:59

   So if you make it harder for them to adopt, if you make it harder for them to understand, it's going to take longer for us to be ready. So right now, we're saying, go use the best model you can find, the easiest way to use it, so you can understand and can we validate the things that these things can do, whether it's from an efficiency perspective, a revenue perspective or a product development perspective that's kind of i'd call it phase one but in phase one what we've discovered is there's different forces for different courses right what people don't understand you know there was a moment whenever we thought the cyber security was going to get consumed by ai models and you know our lives are over because mythos is going to solve every problem well guess what we inspect all traffic at an endpoint at the cost of 20 a year

4. 10:45

   hundreds of megabytes of traffic per endpoint at the cost of $20-$25 a year because machine learning is running on your endpoint, your compute, and we run it on a 20, 30, 40 megabyte payload. Now, if you want to take all that traffic, run that through OpenAI using tokens for every employee in your company, sounds like the sound of revenue. Like OpenAI would probably generate all the revenue in one year from one company. So that's not a practical example of how a model is used.

5. 11:10

   So what you do is you take a small language model, you compress it, you make it task-specific, you keep training it until it gets to 99.9% accuracy and say, instead of using machine learning algorithms, we're going to use generative AI, small language models, and they're going to do that at the end. So my point is, what we're doing is we're trying to figure out what task requires what level of intelligence, and what is the cost of that intelligence for that particular task.

6. 11:35

   And you'll suddenly discover you don't need a Ferrari for every task. And by the way, the customers don't want to pay for a Ferrari for every task. I'll give you the flip side. Flip side is, let's assume you're undergoing a breach. You're being attacked. You know you're being attacked. At that point in time, every customer in the room says, don't worry. You, the general model, need to stop the breach because they know the business impact of that breach in the billions of dollars.

7. 11:57

   At that point in time, I can run agent swarms. You can do all kinds of stuff that you want from a defensive perspective. So there, I may want to use the most up-to-date frontier LLM and say, go think about it, reason the hell out of it and figure out how to stop this breach, how to make this happen. So what we're discovering is you're going to end up using frontier LLMs in unique cases where the business value that is going to get disrupted justifies. You can do drug discovery if you want. I think for most practical use cases in enterprises, we're going to end up training some version of an open source model. And I think when it comes to even in peacetime for us for example running a defender model we most more than likely going to take an open rate model use our own proprietary data train it manage the

8. 12:38

   weights because that sort of gives us the auditability the explainability and the the open system we can actually explain how that happened I completely agree but I'm going to take a slightly different lens to get to the same exact end conclusions let's take economics so So today, $20 for a million tokens for probably generation minus one, 50 if you want to go to the most extreme. We could take one of the Chinese open-weight models, take a Kimi K3, something 800 billion parameters.

9. 13:09

   And if you're on your own GPUs with a little bit of optimization, but I'm talking like a week of engineering work, we could drop the cost to about $1 to $2 for a million tokens. Which enterprise in the world will not accept a 10 times, or in this case, a 40 times cheaper answer for effectively what can do 95% of the work so I think economics is important the second is going to be at the end of the day you just talked previously about chain of thought not just the end tokens well effectively then you're giving away your IP if that chain of thought can be captured and logged and audited by somebody else so so brand and IP is going to drive people to say, I need to run the models on my own. Well, you can't run a frontier model on your own because none of them will give it to you. So then you're back to open weight. And the third is

10. 13:57

    geopolitics. The best open weight models today are Chinese. By the way, I think there's very little danger if you're putting them in a sandbox and running them by yourself. Sure, if you use them from a cloud, that's different. But I think that over the next year or two, I'm very doubtful that there will not be great models that come up in the West that are also open weight and that are as capable as the Chinese. The gap in these things is 6 to 12 months at the most. And so all those three drive me to conclude. I don't know whether it'll be 75 to 90%, but that's the range I would put of all token users will go to open-weight models for the reasons I mentioned, add it to the reasons that Nikesh mentioned. And the token consumption that's left over, is that because you actually need it or is that because it's a default behavior and people are too lazy

11. 14:44

    to change okay so i'm going to talk about developer behavior and since i was one i'm making fun of myself in a younger age developers are lazy they're going to use the most expensive the most expensive answer unless you put a control on it right because it's easy yeah and so that is going to take it there so i would say if experimentation phase you allow that to prove it works and now i got to put on a cheaper answer it's probably fine if it gets me there a lot faster But doing it for production usage is where I am skeptical that it's of value, except for a tiny number of use cases.