LLMs look more like operating systems than utilities

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 8:00

   LLMs don't only have properties of utilities. I think it's also fair to say that they have some properties of fabs. And the reason for this is that the capex required for building LLMs is actually quite large. It's not just like building some power station or something like that, right? You're investing a huge amount of money, and I think the tech tree for the technology is growing quite rapidly. So we're in a world where we have sort of deep tech trees, research and development secrets that are centralizing inside the LLM labs, but I think the analogy muddies a little bit also because, as I mentioned, this is software, and software is a bit less defensible because it is so malleable, and so I think it's just an interesting kind of thing to think about potentially. There's many analogies you can make, like a four nanometer

2. 8:48

   process node maybe is something like a cluster with certain max flops. You can think about when you're using NVIDIA GPUs and you're only doing the software and you're not doing the hardware, that's kind of like the fabless model, but if you're actually also building your own hardware and you're training on TPUs if you're Google, that's kind of like the Intel model where you own your fab. So I think there's some analogies here that make sense. But actually I think the analogy that makes the most sense perhaps is that in my mind LLMs have very strong kind of analogies to operating systems in that this is not just electricity or water. It's not something that comes out of the tap as a commodity. These are now increasingly complex software ecosystems, right?

3. 9:26

   So they're not just like simple commodities like electricity. And it's kind of interesting to me that the ecosystem is shaping in a very similar kind of way where you have a few closed source providers like Windows or Mac OS, and then you have an open source alternative like Linux. And I think for LLMs as well, we have kind of a few competing closed source providers, and then maybe the Lama ecosystem is currently like maybe a close approximation to something that may grow into something like Linux.

4. 9:54

   works. Again, I think it's still very early because these are just simple LLMs, but we're starting to see that these are going to get a lot more complicated. It's not just about the LLM itself, it's about all the tool use and multi-modalities and how all of that works. And so when I sort of had this realization a while back, I tried to sketch it out, and it kind of seemed to me like LLMs are kind of like a new operating system, right? So the LLM is a new kind of a computer. It's kind of like the CPU equivalent. The context Windows are kind of like the memory, and then the LLM is orchestrating memory and compute for problem solving using all of these capabilities here. And so definitely if you look at it, it looks very much like the operating system from that perspective. A few more analogies,

5. 10:38

   for example, if you want to download an app, say I go to VS Code and I go to download, download, you can download VS code and you can run it on Windows, Linux, or Mac in the same way as you can take an LLM app like Cursor and you can run it on GPT or Cloud or Gemini series, right? It's just a drop down. So it's kind of like similar in that way as well. More analogies that I think strike me is that we're kind of like in this 1960s-ish era where LLM compute is still very expensive for this new kind of a computer and that forces the LLMs to be centralized in the cloud and we're all just sort of thin clients that interact with it over the network and none of us have full utilization of these computers and therefore it makes sense to use time sharing where we're all just, you know, a dimension of the batch

6. 11:28

   when they're running the computer in the cloud. And this is very much what computers used to look like during this time. The operating systems were in the cloud, everything was streamed around and there was batching. And so the personal computing revolution hasn't happened yet because it's just not economical. It doesn't make sense. But I think some people are trying, and it turns out that Mac minis, for example, are a very good fit for some of the LLMs because it's all, if you're doing batch one inference, this is all super memory bound, so this actually works.

7. 11:57

   And I think these are some early indications maybe of personal computing. But this hasn't really happened yet. It's not clear what this looks like. Maybe some of you get to invent what this is or how it works or what this should be. Maybe one more analogy that I'll mention is whenever I talk to ChatGPT or some LLM directly in text, I feel like I'm talking to an operating system through the terminal.

8. 12:19

   Like it's text, it's direct access to the operating system, and I think a GUI hasn't yet really been invented in like a general way. Like should ChatGPT have a GUI? like, different than just the tech bubbles. Certainly some of the apps that we're going to go into in a bit have GUI, but there's no, like, GUI across all the tasks, if that makes sense. There are some ways in which LLMs are different from kind of operating systems in some fairly unique way and from early computing.

9. 12:49

   And I wrote about this one particular property that strikes me as very different this time around. It's that LLMs, like, flip, they flip the direction of technology diffusion that is usually present in technology. So, for example, with electricity, cryptography, computing, flight, Internet, GPS, lots of new transformative technologies that have not been around. Typically, it is the government and corporations that are the first users because it's new and expensive, etc., and it only later diffuses to consumer.

10. 13:19

    But I feel like LLMs are kind of like flipped around. So maybe with early computers, it was all about ballistics and military use, But with LLMs, it's all about how do you boil an egg or something like that. This is certainly like a lot of my use. And so it's really fascinating to me that we have a new magical computer, and it's like helping me boil an egg. It's not helping the government do something really crazy like some military ballistics or some special technology.

11. 13:42

    Indeed, corporations or governments are lagging behind the adoption of all of us, of all of these technologies. So it's just backwards. And I think it informs maybe some of the uses of how we want to use this technology or like where are some of the first apps and so on. So, in summary so far, LLM labs, fab LLMs, I think it's accurate language to use, but LLMs are complicated operating systems. They're circa 1960s in computing, and we're redoing computing all over again, and they're currently available via time sharing and distributed like a utility.

12. 14:13

    What is new and unprecedented is that they're not in the hands of a few governments and corporations. they're in the hands of all of us, because we all have a computer, and it's all just software, and ChachiPT was beamed down to our computers, like to billions of people, like instantly and overnight, and this is insane. And it's kind of insane to me that this is the case, and now it is our time to enter the industry and program these computers. This is crazy. So I think this is quite remarkable.

13. 14:39

    Before we program LLMs, we have to kind of like spend some time to think about what these things are,