A severe AI incident could force Washington to seek a slowdown

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 1:13:45

   is events have a way of imposing realities on us that no amount of argumentation, even investigation, can surface. And so what I mean by this is we had our mythos moment and suddenly AI became a national security issue. And like all the arguments that I had previously I no longer had to have, everyone just kind of accepted a new premise. You know, the Hug and Face incident did the same thing with the idea of rogue AI, which, you know, I'm old enough to remember when people like Garry Tan in particular, who's, I will use him as like my whipping boy just for the moment on this one.

2. 1:14:22

   Now, where is, tell us who he is. So he's the president of Y Combinator. So this was, we were a Y Combinator startup. Which Sam Altman once was, and you were supported by Y Combinator. It's a startup incubator. Yeah, that's right. So it's like, it's kind of the, in a way, it's the mecca for startups. Like when you're getting started, the way you know you're a serious founder in Silicon Valley is like, well, now there are other ways, but like back in the day, YC, getting into YC, you know, is this like kind of big thing that meant that your mom wouldn't ask you what you were doing with your life while you ate ramen and made, you know, $20,000 a year or something, right?

3. 1:14:58

   So it was like this thing that said, you know, there's this thing that has an acceptance rate that's lower than Harvard, and they incubated, you know, Airbnb, Dropbox, you know, Replit, like some of the biggest companies, Instacart, you know, the list goes on. So now let's get back to dissing Garry Tan without dissing Garry Tan, which I think was your stated mission. So Garry Tan is now the president of Y Combinator.

4. 1:15:21

   He is a notorious AI risk skeptic, has been poo-pooing the idea of loss of control for as long as he's had a platform. And there was a clip recently, he was on TBPN, saying, hey, look, I think we've got to stop talking about these speculative, stupid science fiction-y risks and focus on the real and present issue of rogue AI agents' swarms taking over our data centers and holding them for ransom. And I was like, I want to take you through time to like last year or five minutes ago and just have you debate the old version of you because- I don't think that would like be, you know, I don't think you guys would be in agreement about what's what's realistic.

5. 1:16:05

   So all of which is to say, wow, that's amazing that he said that. Yeah, because it's like, how many rungs is that from Yudkowsky land? Gary, that's like one. I mean, absolutely. Yeah, I mean, like at a certain point, as you might say, right, we're haggling over price here. Like like this is this is not a serious. And yet, of course, Gary is convinced that his new waterline is for some reason going to hold better and more firmly than all the previous ones he's seen before. And this happens over and over and over again. David Sacks, you know, like, you know, like these things are not supposed to happen and then they happen. And there's a moment of almost intellectual clarity until there's some new narrative that inevitably surfaces that allows us to reinterpret what just happened in a way that allows us not to have to question our identities or egos or anything.

6. 1:16:51

   And so I think this is like a really like important process. Every time this happens, every time the waterline rises, we do gain more converts than we lose. But still, I think on the whole, it's like way too sticky. Like our prior beliefs are way too sticky. And I'm not saying that as somebody who has any special claim to being otherwise. But anyway, so I think events, in other words, just like will force things to happen on their own without advocacy. One of the events that I think is forced to happen is this moment where Trump turns to Xi and says, we got to find a way to stop this. But this is not a moment of friendship. And I think this is like kind of the key distinction, maybe, you may disagree with this characterization, but in terms of the frame that I think will actually be productive, in fact, early on.

7. 1:17:39

   The history of U.S.-China and U.S.-Russia dynamics, but U.S.-China dynamics, has been one of profound mistrust and justified mistrust. If you look at, you know, China's nine-dash line stuff, there's stuff with the WHO, COVID, like, you know, the list goes on and on. China does take a deeply cynical view of really the entirety of the rest of the world. And the idea that we'll be able to trust them without, we'll have to have verification technologies to kind of back up any agreement like this ultimately. But in the near term, what's going to happen is we don't have verification technology. So one of the things we've been spending basically all our time on, or a lot of our time on in the last six months, has been talking to and diving deep on the technology that would allow us to have a trustless agreement with China. So something like what the

8. 1:18:32

   IAEA does, but for AI, but in a way that works more robustly. There are a bunch of technologies for this. One of the things that I think David Sacks has catastrophically failed on is engaging with the fact that those technologies actually exist and have concrete roadmaps that sound like six months, 12 months, depending on what you look at. You could get started on those. He sort of dismisses this, like there's no possible way of verifying adherence to an AI treaty implicitly today. And so therefore, let's not even try. And that second part, I think, is almost like a criminal lack of agency. It's not how any good startup founder thinks about any problem. Like David Sacks himself, if he's worth his salt as a VC, which there are a lot of people who have very mixed views of him in that space. But if he were worth his salt as a VC, he would be saying

9. 1:19:19

   stuff like, okay, cool, that's a problem that you should be able to make a Decacorn out of. This is going to be getting trillions of dollars of value. So obviously you're in a position to capture that value, build a startup to do it. But we don't hear him say that. We hear him dismiss the idea entirely, which I think is just like a catastrophic failure of imagination and just like very un-libertarian of him in some of the worst ways. And so that's part of my gripe with David tax. But ultimately, I think in this moment of crisis, we get our AI Chernobyl event. And well, the president personally feels at risk. This is like one ingredient of it. The waterline has risen to the point where the president...

10. 1:20:00

    family, no one truly feels safe, no one in Congress, because we have now demonstrated that loss of life or the destruction of property to the tune of, you know, many hundreds of millions or billions of dollars is now possible, and that the beatings will continue until morale improves. It now becomes undeniable that we're headed for that world. In that world, the first impulse is to be like, all right, we got to stop this. And again, if you were imagining that this AI Chernobyl event, as I've sketched it out, or failed to sketch it out, is not of a significant enough degree that it would lead to this reaction, because we're literally talking about holding back the engine of U.S. economic growth, would never happen. I invite you to just increase the water line another 10x. Keep doing that until you get to the point where the president feels this way

11. 1:20:40

    and Congress feels this way. Hopefully we're all still around at that point, but that is the moment that I'm talking about. So in the same way that rogue AI agents were science fiction before, in the same way that Mythos was science fiction before, in the same way that ChatGPT was science fiction before and AI agents, the Chernobyl moment now feels like science fiction. I would contend it trivially drops out of taking the very same scaling curve seriously that we have been riding all this time. So we get to that moment. The president says this needs to stop. It needs to stop first and foremost domestically because plausibly, not definitively, but plausibly that