DeepSeek’s headline training cost leaves out its compute investment

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 19:05

   this? I mean, I think there are things that are true about the story and then things that are not true or should be debunked. I think that let's call it true thing here is that if you had said to people a few weeks ago that the second company to release a reasoning model along the lines of O1 would be a Chinese company. I think people would have been surprised by that. So I think there was a surprise. And just to kind of back up for people, you know, there's two major kinds of AI models now. There's kind of the base LLM model, like ChatGP 4.0, or the DeepSeek equivalent was V3, which they launched a month ago. And that's basically like a smart PhD, you ask a question, gives you an answer.

2. 19:48

   Then there's the new reasoning models, which are based on reinforcement learning, sort of a separate process as opposed to pre-training and 01 was the first model release along those lines. And you can think of a reasoning model as like a smart PhD who doesn't give you a snap answer, but actually goes off and does the work. You can give it a much more complicated question and it'll break that complicated problem into a subset of smaller problems. And then it'll go step-by-step to solve the problem. And that's called chain of thought, right? And so the new generation of agents that are coming are based on this type of idea of chain of thought that an AI model can sequentially perform tasks, figure out much more complicated problems. So OpenAI was the first to release this type of reasoning model. Google has a similar

3. 20:39

   model they're working on called Gemini 2.0 Flash Thinking. They've released kind of an early prototype of this called Deep Research 1.5. Anthropic has something, but I don't think they've released it yet. So other companies have similar models to O1, either in the works or in some sort of private beta. But DeepSeek was really the next one after OpenAI to release the full public version of it. And moreover, they open sourced it. And so this created a pretty big splash. And I think it was legitimately surprising to people that the next big company to put out a reasoning model like this would be a Chinese company. And moreover, that they would open source it, give it away for free. And I think the API access is something like 1 20th the cost.

4. 21:27

   So all of these things really did drive the news cycle. And I think for good reason, because I think that if you had asked most people in the industry a few weeks ago, how far behind is China on AI models, they would say six to 12 months. And now I think they might say something more like three to six months, right? Because O1 was released about four months ago and R1 is comparable to that. So I think it's definitely moved up people's timeframes for how close China is on AI. Now, we should take the claim that they only did this for $6 million. On this one, I'm with Palmer Luckey and Brad Gerstner and others. And I think this has been pretty much corroborated by everyone I've talked to that that number should be debunked. So first of all, it's very hard to validate a claim about how much money went into the training of this model.

5. 22:26

   It's not something that we can empirically discover. But even if you accept it at face value, that $6 million was for the final training run. So when the media is hyping up these stories saying that this Chinese company did it for $6 million and these dumb American companies did it for a billion, it's not an apples to apples comparison, right? I mean, if you were to make the apples-to-apples comparison, you would need to compare the final training run cost by DeepSeek to that of OpenAI or Anthropic. And what the founder of Anthropic said and what I think Brad has said, being an investor in OpenAI and having talked to them, is that the final training run cost was more in the tens of millions of dollars about nine or 10 months ago. And so, So, you know, it's not 6 million versus a billion, okay?

6. 23:17

   A billion dollar number might include all the hardware they bought the years of putting into it, a holistic number, as opposed to the training number. Yeah, it's not fair to compare, let's call it a soup to nuts number, a fully loaded number by American AI companies to the final training run by the Chinese company. But real quick, Saks, you've got an open source model and the white paper they put out there is very specific about what they did to make it and sort of the results they got out of it.

7. 23:50

   I don't think they give the training data, but you could start to stress test what they've already put out there and see if you can do it cheap, essentially. So I think it is hard to validate the number. I think that if, let's just assume that we give them credit for the 6 million number. My point is less that they couldn't have done it, but just that we need to be comparing likes to likes. So if, for example, you're going to look at the fully loaded cost of what it took DeepSeek to get to this point, then you would need to look at what has been the R&D cost to date of all the models and all the experiments and all the training runs they've done, right?

8. 24:29

   and the compute cluster that they surely have. So, Dylan Patel, who's leading semiconductor analyst, has estimated that DeepSeek has about 50,000 hoppers. And specifically, he said they have about 10,000 H100s, they have 10,000 H800s, and 30,000 H20s. Now, the cost of that- Is they DeepSeek or it's DeepSeek plus the hedge fund? DeepSeek plus the hedge fund. But it's the same founder, right? And by the way, that doesn't mean they did anything illegal, right? Because the H100s were banned under export controls in 2022. Then they did the H800s in 2023. But this founder was very farsighted. He was very ahead of the curve and he was, through his hedge fund, he was using AI to basically do algorithmic trading. So he bought these chips a while ago. In any event, you add up the cost of a compute cluster with 50,000 plus hoppers, and it's going to be over a billion dollars. So this idea that

9. 25:28

   you've got the scrappy company that did it for only 6 million, just not true. They have a substantial compute cluster that they use to train their models. And frankly, that doesn't count any chips that they might have beyond the 50,000, you know, that they might have obtained in violation of export restrictions that obviously they're not going to admit to. And we just don't know. We don't really know the full extent of what they have.

10. 25:59

    So I just think it's worth pointing that out that I think that part of the story got overhyped.