What happens on the day the president fears AI will kill him? | Jeremie Harris (x-post from NonZero)

risks that we've been hearing about lately, you know, bioweapons, cyber, and the phrase the loss of control is in there. Something that has become more prominent in the discourse since the hugging face incident. So are you glad that, you know, regular people have finally caught up with your prescience? You might not put it that way. Well, that's, I mean, I'll accept the framing begrudgingly. But yeah, I mean, I think it's clear that right now the idea of rogue AI is now sort of widely accepted because it has to be, because we've literally seen it. You know, it was, so that report, you know, did come out in 2024, early 2024.

It was the product.

say one thing in anticipation of our conversation uh about where you and i do and don't overlap because i think this is what's uh going to make this an interesting conversation as i've said we we both emphasize the eventual need for something that I would call international governance.

You might or might not. Some people prefer other terms. But one difference between us is I've heard you describe yourself as a China hawk, and I've heard you talk like a China hawk. I've heard me describe myself as definitely not a China hawk. So it'll be interesting to see how we navigate some of these issues together and what we do and don't agree on, on what's possible, first of all, and what's necessary and what the implications of all this are. But yeah, back to the discourse, it has been kind of shocking. I mean, I have, it's funny. So you wrote this report before some of this stuff happened. I put this book of mine to bed, The God Test, before the breakout.

the OpenAI breakout, you know, and the trajectories we're seeing now unfold are in the book, you know. We already had like Malt book and various, and I talked about agents and I talked about loss of control. In fact, I have a chapter called Hive Minds and the Loss of Control. Although I didn't, you know, I didn't anticipate the specifics of this kind of scenario. So I've, I've had this weird feeling of like, wait, so I'm prescient. Oh, wait. So no, it's too late to be, it's like, it's moving. The discourse is moving so fast. And two things. First of all, people in this realm have for some time been saying, look, what's it going to take to get the world's attention?

i guess we'll need some you know a warning shot by which they generally mean meant some kind of high casualty catastrophe yeah in that sense the news is good the over the windows moved a lot very low body count i mean you know there aren't you know there have been tragedies involving ai but uh so far as i know nobody died at hugging face and um and so that's all good but i'm wondering what you think in terms of, first of all, what is the discourse still not catching up to in your view? And secondly, do you think it's still going to take more kind of jolting incidents of some kind or another to get the discourse to where you want it to be? Yeah, I think these are really complicated questions, but I think at least in my view, they have a relatively simple answer. So the simple answer is there is still somehow alpha in just taking scaling seriously. So scaling is this idea, I think by now it's sort of a household idea, but just in case it isn't, just this idea that if you want to make an AI system smarter, all you have to do, roughly speaking, is train it at a larger scale with more computing power and more data. So you take this model, this artificial brain, right? And again, you use more AI processors, more of these chips in a bigger

AI data center with like more power and so on. And now we're at the scale where we're doing this with gigawatt scale. A gigawatt is like a million homes worth of power, right? So these are truly like mammoth, mammoth facilities and sites. AI buildouts are responsible for the lion's share, if not basically all of US economic growth over the last few quarters. So if you zoomed out, really, I mean, you would see planet Earth is slowly transforming itself, not so slowly even into just a computing factory. And the economy is sort of starting to reflect that. The markets are starting to reflect that.

Political pressures are starting to reflect that. And yet, this simple fact that scaling just works is still something that by believing it more than others, you can still, I think, make better predictions than the average person. You know, back in 2020, like the first time that my co-founder and brother Ed and I bet big on this was when we left our company in 2020. There was a paper that came out around that time um it was the first big scaling laws paper was that was that a company you started i'm just curious and what was it yeah so it was an experiment and this may be i don't know if this goes to our our differences necessarily but this was a like hardcore experiment in like libertarian economic theory so um well that would be one difference not necessarily related to the china hawkism issue but uh yeah i don't i i'm looking for milton friedman on my bookshelf oh not there. Yeah, absolutely. So and to be clear, like, you know, as we age, our views become more nuanced. But this was sort of like a fun experiment in like how far could you take that? And so he had this notion of income share agreements where you might get educated at some school or whatever, get some training. And then the training could be monetized by taking a percentage of your salary

for a certain period of time. So you don't have to pay up front. And nominally, incentives are aligned in that way, we set up actually it kind of became the world's largest marketplace for income share agreements. And we would do kind of peer to peer. So if you wanted to break in, typically it was into AI or data science, you'd work with a mentor and the mentor would say, okay, I'll help you for free. You know, there's somebody who works at Tesla or OpenAI and in exchange for like 3%, 5% of your first year salary when you get hired. And so we had like thousands and thousands of these mentorships happening. It was a really cool experiment. And we got some really genuinely fascinating economic data from it that is, I think to this day, quite unique, but the scaling loss paper came out and, you know, I still remember I had a two hour conversation.

So this is, this was, was Dario on this, on this paper or this is, yeah. When, when is this? And this is establishing, arguing that, yeah, all it takes is more compute, more data. And that certainly has gotten us well a lot of innovations have happened let's face it but that has that has partly driven what we've seen was that that was dario's name on that yep that's right so so dario was involved in all the early scaling research in fact he was kind of like the the champion of scaling almost like this it's often said of open ai that they are really scale pilled and and they are but um there there are a lot of ways in which anthropic actually is much more aggressively scale-pilled than OpenAI in different ways. And a lot of that comes down to Dario. He was the earliest believer in it, really, along with Ilya in a lot of ways.

And he was at OpenAI when they first showed the benefits of scaling with the early chat GPTs. That's right. Yeah. And so if you talk to the OpenAI people, so like Jack Clark will tell you, GPT-2 is kind of like the moment that he went, whoa, whoa, whoa, whoa, whoa. And really, like maybe it's worth talking about just like what these things are scaling because it's it's abstract to say you know they're like you scale data you still scale computing power or whatever but like the actual thing that you're scaling is uh basically a text autocomplete engine and so all it does is it gets really good at predicting the next word um it sounds stupid but text autocomplete ends up being by some weird freak of information theory this like really really important task. And you can kind of understand why that's the case. If you think about, say, if I give you a sentence like, to counter a rising China, the United States should blank, right? In order to fill in that blank, in order to fill it in accurately, you need to know what the United States is, what China is, what it means for China to be ascendant. You need to know facts about geopolitics, economics, all kinds of crap that goes into just like filling in that one next

word. And so this turns out to be a really powerful way of teaching AI models just about the world generally, facts about physics, about how to code, about how to orient yourself in the world. And so- I mean, if I could add one thing, you also need, if you're going to do that efficiently and the process of training forces the machine to do it efficiently, you need a way of like, for example, representing the meaning of words. And I think it's underappreciated the extent to which, you know, something that the brain presumably does. I think it's underappreciated the extent to which training is kind of the reverse engineering from language of some of, I would say, some of the cognitive functionality that in humans evolved to help us deal with language. And then, of course, there's the subsequent so-called post-training where they do more than autocomplete. They teach it to answer various kinds of questions and so on.

And I think the reverse engineering continues. And this is why the stochastic parrots people are just not getting the power of what's going on we are without having to understand how the human mind works we're kind of reverse engineering the rough equivalent of it and all you need is uh the right kind of data and a lot of scale yeah that's right like it's and and the analogies are really interesting here. And there's a sense in which, like, it's really easy to get lost in the minutiae, you know, the Gary Marcusing of a lot of the conversation here, where there's always some fascinating reason to believe that scaling will magically stop. And we, you know, perennially, there is some fascinating argument. We're hitting a data wall, we're hitting a compute wall, we're hitting an energy wall, we're hitting this wall, that wall, stochastic parrots, as you said, right? These arguments that often wildly outlive, at least like in the discourse, they wildly outlive their utility or their evidentiary foundation. Like we're way past stochastic parrots. I'm old enough to remember when the data wall was supposed to be a reason that you wouldn't get offensive cyber capabilities like Mythos. Like we just don't, we don't seem to

refactor our kind of credibility index when we're thinking like, how seriously should I take a Gary Marcus. Well, you know, maybe Orien LeCun, maybe we ought to look back and see what predictions they've actually made about the present moment before we go off and, you know, and the economic version of this too, which kind of drives me up a wall a little bit is, you know, people who say, oh, well, this whole AI thing is overhyped. There's a massive bubble. It's going to pop. These aren't real businesses. And while it could be a bubble that pops, and I've heard you argue sort of similarly before on the show, I think this is exactly right. It could absolutely be a bubble the POPs, independent of the trajectory of the technology for any number of reasons that are mostly economic and not technological. But Anthropic has had its first profitable quarters.

Like if they want to, they can turn this into a money machine. The race to scale right now is about who owns more market share. It's the classic kind of VC financing model. And the margins for these companies have been going up. Like if you look at the margins for Anthropic right now, they're sitting increasingly in comfortable software as a service territory, like your 70% territory. Opening eyes headed that way too. And so far from dropping, again, I'm old enough to remember when the margins were supposed to drop, everything was supposed to go dark. This is the opposite of that trajectory. And consistently, everyone has been surprised by how fast things have gone and not how slowly. I say this with some modesty because I called it incorrectly in the other direction where I said, hey, frankly, I thought we ought to be dead by now based on my median estimate. So what the hell do I know? But yeah, I mean, that's the shape of it as I see it. I think we're accelerating faster than most people appreciate.

Yeah. I'm going to add an asterisk, but not pursue it because I want to get back to where I initially derailed this by asking you what your company was before you and your brother moved on to your next venture. I will say, I hope we'll get back to this because it's relevant to the China question and whether you can slow down without China catching up and so on, which is that the challenge Anthropic does face is staying ahead of the commoditization, right? I mean, and the value of what it's doing now lies in the fact that it is clearly, its models are superior are too much cheaper models. And that is, you know, one question mark is, can it sustain that indefinitely? If not, what will plan B be? But anyway, I'm sure you have things to say about that, but I'm kind of feeling guilty about taking you off your original path. We should return to the part of your story, I guess, where your libertarian, one more asterisk, ironically, some of your, I don't want to say enemies, but some people who drive you crazy are libertarians because they're still arguing that this technology needs essentially no regulation.

Another thing maybe we'll get back to, we will not name any names. The name David Sacks will not come up. No, Mark Andreessen will not be mentioned in this. Will not be mentioned, but let me take you. So let me take you back to you. where you were. Continue your journey. Well, yeah. So actually, I guess we've covered part of it. This paper comes out. Ed and I at the time were very happily working away in this weird Milton Friedman-style economic experiment. The paper comes out, and we have a sort of tense two-hour phone call. And he gets the better of me. He's basically like, look, I've looked at this paper. The scaling laws, they seem to be holding across. It's like seven orders of magnitude in computing power. So basically, if you train a model with 10 units of compute, and then you train it with all the way up to, you know, like 10 million units of compute, 100 million units of compute, and you just see this merciless, ruthless, unending straight line that will not bend, it will not yield. And this has a certain aesthetic to it that if you think about how evolution worked, like for evolution to work, you need to have a very smooth hill to climb.

There's got to be, if you're going to work your way up the intelligence ladder through evolution, an increment of intelligence has to yield an increment of reproductive fitness, right? And so it had to be a simple thing. The brain had to be something that was made up of simple components that could gradually increase in complexity over time, over generations. And when you look at a scaling law like this, I mean, the argument Ed was making was like, look, of course nature would be shaped this way. Like, it had to be a scaling law of some kind. We just seem to kind of have unlocked it. Transformers, which is the architecture of the essentially artificial brains, the AI models that we were scaling in this way, just seemed to have the property that they were extremely resilient, that these scaling curves would not yield.

And so, well, this was enough for us to go, when GPT-3 came out, which was, gosh, like, what, two and a half years before chat GPT. We looked at that and we're like, oh my God, like, not only do we have these scaling curves, But now for the first time, we have an AI that, bear in mind, was not ever meant to be anything other than a text autocomplete system. And yet it can code. It can design basic websites. It can translate between languages. It can do all kinds of things it was never meant to do just by virtue of the scale of its training. You train it on enough internet text, enough Wikipedia pages, enough textbooks. Eventually, if you give it a sentence like, below is a piece of code that checks the weather, colon, it will know like, okay, well, I've read a bunch of blog posts online that have a description of a piece of code and below is the actual code. So I'll just write up that code. It could do that. And so immediately to us, this implied we were on a trajectory for human level AI and beyond. Immediately it implied that AI was about to become the single most important source of national security risk in human history. It sounded crazy then. We literally started going around talking to anyone who would listen in the US government and other governments,

UK, Canadian, try to wake people up. And one of the things that happened, and this is something

But I'm wondering, did they have a plan at that point as to how they were going to kind of circumvent normal channels of Washington influence and yet save the world? I mean, it's funny. I think maybe part of the answer to your question is this vibe you still get from people like Dario. I mean, not maybe explicitly, but you get the sense that they think, well, if we just are the first ones to build superintelligence, we will, through alignment or some kind of, I would say, magical process, somewhat dismissively, because I don't think this is going to do it. but through some such process, as long as it's a good guy who does it, as long as it's Dario and not Sam, then we'll be okay. It was like a private sector. That's a very libertarian solution. We don't need the government at all. But was that what they were thinking?

Well, so, okay, this is a really good and important and interesting question. I think one thing I'll just say, we made the choice once we started to see this pattern where we were hearing really confident, and by the way, like these like very confident calls they were making didn't just end at don't talk to government because this will be the response. They also included just like massive miscalculations on AI timelines and not necessarily, okay, so AI timelines and also what you do about them. So I had many arguments with folks in this ecosystem that like, look, scaling laws, I mean, I think this is the trend that gets us there to superintelligence. And there was all this sort of like speculation at the time about, you know, actually Ajay Khotra, who I'll just name because I'm actually a huge fan and she's over at Meter now, was involved in. She's very impressive. Yeah, that's right. Well, at the time, she kind of championed this bio anchors framework, which was a way of thinking about AI progress through the lens of like, well, here's how much time or how much compute, I should say, how much effective like kind of computation was involved in shaping the human brain and kind of leading to

evolutionary intelligence. And it turns out when you look at evolution as the driver of intelligence, and you think that that's how we're going to get there with AI, you end up with these very long timelines. So you end up thinking, you know, oh, let's like wait, you know, 2040, whatever. And because this ecosystem is very, it's still actually very insular in many ways, a small number of these kind of influential assessments steer the ship in ways that would be quite surprising. And so we found ourselves wrestling with this framework that which, you know, Ajay has since updated away from and, you know, OpenPhil, now Coefficient Giving, has updated away from. But this was, I think, a really important thing, because if you actually believed in short timelines, it would cause you to act. And that action in.

would turn out later, I would say today, to be really valuable pre-positioning. It just takes a really long time to get to know the intelligence community, to get to know the Department of Defense, Department of War, to get to know all the stakeholders on the export control side of the house, to put together the pieces of the puzzle that you will need at game time. And if we had sat back resting comfortable with this notion of sort of 2040 AI timelines, we would not have done any of that and and in particular we actually never would have done that state department assessment which um like i don't want to overplay it but like it a lot of the whistleblowers we spoke to uh since and very shortly after went public having seen the result of you know actually we can have this conversation in public like the the media is not crazy about this uh or on this issue and so on and so i think that there's a sense in which like you can talk yourself into doing nothing um this is advantageous particularly if you are one of the small set of people who are authorized implicitly by this ecosystem to be agentic to to behave with agency to do to make the first move to tell people when it's okay to do x y and z um and so so anyway i think that was

like a a big a big challenge in that ecosystem um we uh yeah we definitely so we ended up turning down money from all effective altruist source. Like we have to this day taken no grants from anybody simply because we suspected that at some point this might actually become political and we wanted to make sure we were clean. That, by the way, was an extremely painful thing to do. It meant literally standing up an entire for-profit business that would subsidize our crazy unprofitable work trying to first understand and then intervene in the development of AGI. But it did mean that that we, for a long time, were essentially the only independent people, as far as I know, with the exception of maybe folks like, you know, Zvi, who will write whatever the hell he wants to write and thinks. But, you know, by and large, in terms of the entities involved in this ecosystem, they were all funded by this kind of group of funders, which is not like intrinsically a bad thing. It was the only money chasing this problem. Like open philanthropy, coefficient giving were incredibly prescient about this. It's easy to like get into, you know, open fill bashing. And I'm happy to indulge in that. But like, I think fundamentally they got it much more

right than almost anyone who's waking up. No, I mean, I was going to ask you, and this is a good transition. When you use the term loss of control in your report, do you have in mind the full on Yudkowsky sci-fi sounding scenarios, because you do have to give this crowd credit for impressions. I mean, it hasn't happened yet. And I personally think that, well, my P-Doom is lower than Eliezer's, but the stuff that sounded crazy is now being taken seriously by sober people. and i just you know i uh i was aware of eliezer vaguely for many years had him on the podcast 15 years ago i i i was i was not entirely dismissive but i was a skeptic and i i just when you think about the when you look at the state that ai was in at the time they started saying this stuff right uh it's damn impressive hats off and that goes for the whole culture in northern california they were part of whatever its eccentricities uh well but go ahead so there is this phenomenon right where um when when you when you make a prediction that's really far out and and the world looks completely discordant with that prediction but you have some sorts of alpha that makes you think that's a correct prediction um you you tend to think of like the

the, the thing you're predicting. as this big giant peak, right? So back in 2005 or whatever, Yudkowsky's like, you know, rogue AI foom scenario, we're gonna lose control over it. That's what AGI looks like. And that looks like one thing. If you judge him on that basis, he was right every step of the way, right? But as you get closer to the peak, you sort of find that it resolves into separate peaks. It's actually a more complex landscape once you get close up to it. And then, okay, we- By which you mean, I mean, flesh that out like so you see what peak and what peak instead of just one peak well yeah so so you know one version of this is uh eliezer at the time predicted that neural networks were not going to be the path to hei and so you know you can you can you can say ah what a dumbass like you know failing to call neural networks that's the single most important prediction of that well was it though or was the prediction that actually once you build close to human level ai you're going to start to kind of not be able to control it because that was pretty science fictiony sounding thing to say at the time at a moment that predated any notion of what the actual architecture would

be neural network or otherwise it would lead to this so you know like you you can you can keep sort of rerunning this test but as you say i think from from a distance and in many ways closer up um yeah he's he's done really well there are important ways i think in which the world has ended up looking um so so i want to be careful here because what you'd kowski would say is he has been careful not to say that there would be a loss of control event of any particular shape before the kind of ultimate moment, that there would be certain kinds of warning. He's got a pretty, on paper, even-handed kind of probabilistic approach to this. And broadly, everything that's happened is consistent with his very uncertain model. He's not quite committed to any particular way things unfold, which is probably right overall.

all. But at the same time, it makes it tempting to say stuff like, well, did you predict this thing? And then it's easy. It's too easy to say, well, no, he didn't. But it's not that he didn't. And I would say, as for missing neural networks, the whole value of his approach is that it's methodology independent in the sense that he's looking for general laws of goal-seeking intelligence, kind of. It doesn't really matter which approach winds up working out. he's already in a way, the most important thing about his argument is that it doesn't matter which methodology leads to this. Yeah. And he's made other kind of side quest arguments that do, you know, to varying degrees, depend on things like that. But, but the big picture exactly to your point is that, and, and, you know, so this is, this gets us back to, you asked this question earlier about, um, you know, making predictions and, uh, you know, what, like, in what ways is the discourse right now not caught up to what's coming? And, you know, I pointed to taking scaling seriously. And like, you know, there's this interesting question of what happens if you take scaling seriously? What happens if the same party trick that we've built a minor career on

over the last six years just continuing to say, nope, like every time somebody comes up with an interesting sounding objection, nope, that will melt in the face of further scaling. I don't know how, but I don't need to know how, the combination of human innovation, massive economic incentives, and just the incredible flexibility of neural networks and the optimizers we use to train them and the training mixtures we use to train them will just dissolve this objection in due course. That trend, if you believe it, it will continue. I think you continue to get shocks that are surprising to a lot of people. And there are other scaling curves like that or other really robust curves think here about things like moore's law right like any anything that that has that that kind of logger or exponential characteristic um that that is really robust over a long period of time if you extrapolate it there's often a lot of alpha in it because people often just can't accept that the world will look like it will with 10 times more of the thing in it. COVID was a great example. You had scaling curves for COVID. There were just these contagion charts, basically. And if you took it seriously, you would have shorted the market in early 2020, as many people did. Many people, by the way,

in that same effective altruist, Yukowskian, rationalist ecosystem, they're owed an awful lot of predictive points for that. We don't get to just discount. The people who called COVID and AI, if you're a person who called neither, I think there's a certain amount of humility that you have to kind of demonstrate when you're assessing those arguments that we all have to. But yeah, so I think if we continue that scaling curve, one of the things that pretty clearly starts to happen, apart from more hugging face incidents and more mythos for bio incidents, and increasingly like what does this look like for robotics and drones, you get to a point where the waterline rises and rises and rises. And I do think you get an AI Chernobyl event, unfortunately. That's kind of what we've taken to calling it. I'm, in a sense, not going to bother to define it. This will sound like a kind of cheap out, but I think this is the right way to do it. Whatever this AI Chernobyl event is, my definition of it, our definition of it, is it's an incident of great enough severity that it leads to a sea change in the official position of the US government on AGI risk. In particular, it leads to a view that we must now regulate,

slow down, pause in some way, the development of AI domestically, which then necessitates engaging with China. There's just no two ways about it. And so whatever the threshold is, and I often get into these circular conversations with people who will say, oh, well, you know, do you really think that the US government will do something about it, will change their policy on it um with just like you know a thousand deaths and and i'm happy to go okay well just increment it more and eventually we get to what i would define as ai chernobyl is defined with respect to the response of the u.s government to the incident in the same way that i would define super intelligence as that thing which if you build it you die that that's i think the most actually useful definition of super intelligence um which i'm what i'm not saying is now do you actually you buy that part of the yudkowski thesis and we should say we mean super intelligence in the traditional sense of the term yeah not not uh not in the the trump anointed sense of regular artificial intelligence i don't i don't want to get into how frustrating i find it that you can throw this big a wrench so gratuitously into the discourse right because we have these

terms that mean something and everyone who knows about this shit is using them in this one way but i said i won't get into that anyway you mean super intelligence and you kowski since it's vaguely defined in inherently maybe although maybe you think you have a very crisp definition of it but do you do you agree that maybe i misunderstood you but the that if we don't stop the the progress before that point, we are very likely annihilated. Yep. To be clear, I'm not like a, you know, Roman Jampolsky style. I'm going to give you my PDM is like 99.9999999999% or whatever. But but so, I mean, the facts of the matter at this point are, you know, we have no there. I think

punctuation point to it and then move on to something else. The punctuation point is, look, I haven't looked into the dates exactly, but I have a hypothesis about Sam's famous declarations a fairly long time ago that this is a huge problem. There are these Yudkowsky-ite quotes. I would conjecture that if you look at when he uttered them, it was when he was courting Elon Musk and trying to get Musk to invest in OpenAI, knowing that Musk himself was prone to these kinds of vision. Again, it lines up roughly, I know, if you just look at the year or two when he was saying the most dramatic forms of this, but I haven't looked into the exact dates.

the podcast. I don't want to cut you off there. I want to let you finish. I want to say a couple of things. One is that, folks, if we stopped here with this level of AI, I mean, if even the publicly released stuff, let alone what's sitting in the closets in these companies, and just let the application of it filter through the economy and the personal lives of human beings, there would be plenty of disruption in both the good Silicon Valley sense of the word and the bad sense of the word, okay? So don't talk to me about choking off productivity gains and stuff. We'll be okay. And I personally think that disruption in the bad sense of the word would be about as much as we can handle if we did stop things there. And God knows if we stop things with the models in the closets, right? That's enough. That's enough, God. I mean, could you give us a break uh i don't know if you i i think the economic argument there is kind of interesting so if you look at um uh what so an awful lot of the economic impact here comes from not just the fact of having models but the the infrastructure build out that undergirds this

right so well there's that yeah it's like the aircraft carriers worth of capital that are being you know outlaid to build the next 50 billion dollar data center um you know there's this argument and sort of, I think we talked about it quite a bit on like last week in AI, but I don't think people fully appreciate the extent to which interest rates are at risk of going up quite a bit due to the higher borrowing costs that are going to come from, because like right now, more and more of the infrastructure build out is being financed by debt. And semi-analysis has a great, you know, Dylan Patel and Dwarkesh have a great podcast on this.

You can check that out. But the analysis behind it is pretty damn sound. You're seeing borrowing costs at Meta north of like 5%, pushing 8% potentially in the next couple of years. If I can give Meta my money and they will give me a return that's 8%, well, eventually that's going to drive up borrowing rates across the economy, especially given that this sector can absorb so much capital. I mean, it's just like you can find a way to put a trillion dollars and still make that kind of return on it which is insane it just hasn't happened in any other context in human history before and so um i like i think the the impact of stopping the build out the infrastructure build out itself rather than just finding new productive ways of serving the models um would would probably be very dramatic uh maybe you could keep the build out going if you just use that infrastructure to like serve the models and not train them like you know there's there's some jiggery wouldn't shock me wouldn't it would eventually catch i don't think it would be long before you caught up in that regard with the capacity. I agree. I mean, it would be disruptive, but, you know, so these are two different things we're talking about. There's

the anticipated productivity growth, which I think would keep, you know, you would see some of that for some time to come if you froze them all now. And then there's what you're saying, which is the abrupt halt in the infrastructure spending. And I personally would say that that That little jolt might save you bigger trouble down the road. Who knows, right? Because that kind of bubble can burst at any point. And the bigger it is, the worse. But we're getting into the maybe economic weeds. I'm curious what you think. And then I will do some sort of segue into the overtime segment, but go ahead.

No, I mean, so I think I'm personally, I won't say skeptical of the bubble burst. I just think that the actual productivity gains coming from these models and from that infrastructure build out are such that they actually do compensate for the cost of it. So again, Anthropic having their first profitable quarter shows us that we are there in some sense. At any given time, Anthropic has to make the call, do I invest more today into building a data center, which by the way, will only come online 18 months from now. So they're kind of doing this uncomfortable projection thing where they're like, if I under invest in compute today, then 18 months from now, I'm actually going to, well, oddly, I'll find myself like flooded with more inference requests than I can handle and forced to therefore make smaller training runs.

And then, you know, there's kind of that balancing act. Whereas if I overinvest, then what I can do, as SpaceX showed recently, is I can basically just like rent out my compute to whoever has the leading model. And there's this really interesting economic effect where now, essentially, you've got all this compute infrastructure that gets built out at risk, but it can always just be repurposed to serve whatever the best model is. So like you have this giant pool of compute that's just like chasing whatever the leading model is at any given time with the Frontier Lab that builds that model taking the larger margin cut.

And so I think, you know, in that sense, there are a lot of ways for the bubble to kind of be softened on the other side. It could still burst. It absolutely could. I just think that there's like a pretty plausible economic argument for the returns being there to justify it in the long run. okay so here is my uh setup for actually moving into the overtime segment which will be available to paid subscribers non-zero newsletter link in the show notes or go to non-zero.org um which is uh to kind of change the subject and and get back to what i claimed we were going to talk about at the beginning and we will but uh this was what you had to say about all this stuff was so fascinating. I wanted to stick with it. So here's a contention of mine that may evoke protests from you, and it implicitly gets at the China hawkism issue. But my contention, if you look to get back to your report, your 2024 report, and the various kinds of risks that I think are real and plausible from AI, ranging from some form of loss of control to the, you know, bioweapons, cyber, and just the possibly destabilizing effect of rapid AI progress and power shifts, you know, and so on, geopolitically as well as intranationally,

I would say. I think there's a case, you know, for slowing down and for suppressing specific illicit uses. And my contention is that I agree with you for sure. That's ultimately a thing that has to be addressed internationally. And I would say my contention is that, you know, as is recognized by many people, this is going to be more complicated than arms control with the Soviets during the Cold War. And one way I would put it is to say that the degree of transparency we're going to have to have around the world, in other words, the extent, the kind of fine-grained extent to which the United States and any nation is going to need to be reassured that their worst spheres are not materializing in some lab.

some other country. When you look at that degree of transparency, we are going to eventually need a revolution in international relations such that we, for the first time, become a true global community and aren't having these big wars and are not on a fundamentally adversarial relationship with, say, China in particular, because, you know, kind of walking and chewing gum at the same time as you did the Soviets, you know, deep hostility, very adversarial, you know, we're staging coups around the world, but still we can agree on, you know, anti-ballistic missile treaty and so on. I just don't, I don't think that's a world we're moving into.

So, you know, I think we're ultimately going to have to get to a place that strikes people as super idealistic and a place where even you are no longer a China hawk because our relations with China are that good. That's my long-term vision. I'm excited to disagree about this productively. But good, good. Well, you know what? But people are going to have to pay to see it because we're going to move into overtime. But first, I got to say again that people should check out your podcast, Last Week in AI. Really interesting chemistry. It's funny. Chemistries between co-hosts are coming in many varieties.

This one is distinctive. Very, I would say, sharp division of labor between the two of you that works out very well. And, you know, I'd say, well, I'd say more specifically, there are a lot of AI podcasts that have people like you, Andrew Coase, who know a lot about AI. and very often that makes it just entirely inaccessible to even a fairly well-informed non-technical person. But you, yourself, this is part of the division of labor, is that you go into, you know, you remain mindful that not everyone is already on the exact technical wavelength i mean i remember your explanation of uh sparse auto encoders for example um years ago possibly uh and so anyway it's uh uh my hat's off we need we need more accessible um ai podcast i mean i like when i get to talk to people like you that emerges but there aren't there aren't many podcasts where the regulars are technical people and yet it can it can be profitable for a lay person to listen in so i i really appreciate that thank you thanks bob and now uh if there's anything else you want to say like what's your twitter handle uh oh yeah um i'm pretty sure i'm pretty sure it's my first name so j-e-r-e-m-i-e c harris h-a-r-r-i-s i'm pretty sure that's

it but if you look up my name you should be able to find my my twitter pretty quick um i will put in a plug for, uh, so we just, um, uh, came out with this, this, uh, website and, and sub stack, uh, AI end game.com. Basically, this is us trying to figure out, um, you alluded to it, but just like, if you are now, wait, us as you and your brother or you and your co-host, I mean, my brother. Yeah. Yeah. Yeah. That's right. So, um, there, there's kind of this thing that's happening right now and it's, it's been happening for a while, but like there's a community of people who are AI safety hawks who are really concerned about, you know, losing control of advanced AI. And then there's a community of people who are China hawks. And typically what happens is the loss of control people are so, I think, rightly concerned about loss of control over AI, so rightly concerned that we have no clue how to begin to control these systems that we're about.

build that they um reflexively go okay like we we need some kind of international uh kumbaya moment with china um and and and therefore we need to interpret by the way when you use that word kumbaya on your podcast or anyone does i i i uh i get this this uh primitive aversative response uh uh uh wells up in in me we'll get into that later all right then i'll modulate my language but uh Basically, I'll put it this way, concretely, that we should interpret, for example, the CCP or Xi Jinping's own words about AI control in the most sort of sincere and charitable way as we can, because the prospect of us going head to head with an adversary as capable as China with the stakes associated with superintelligence just seems so insurmountable.

On the other hand, the China hawks kind of do the same thing. They go, well, you know, classic example is like the diplomats that we've been talking to quite a bit who have done U.S.-China, who sat across the negotiating table from China for a whole bunch of these international agreements, and, you know, they'll be like, like, China, forget it. Like, it's just not going to happen. You're not going to get any kind of agreement. They are, you know, hostile and full of shit. They are, in my view, completely correct about that. And yet they reflexively go, and therefore, because my pet problem is unsolvable, this alignment thing must be a hoax. It must be solvable in some way. Just give them more time. They don't understand that the shape of the alignment problem is fundamentally different from the shape of the problem of scaling intelligence in the first place.

And so this is kind of our attempt to bridge the two, to have a coherent view of the AI endgame that both stares in the face what we believe to be the realities regarding superintelligence and loss of control and also China, and to answer questions like, you know, what do you do when you genuinely are just six months ahead of a nation-state adversary and you're building a technology that leads you off the edge of a cliff? And so I'm not saying that we have the right answers, but I'm just hoping that it's a kind of intellectually honest attempt to hold both thoughts in our head at the same time.

All right. So we will possibly explore that very tension in overtime. In any event, we're going into overtime. Thanks to everybody who's followed us this far. Hope you'll join us in overtime. OK, why don't we start with just your response to that question? My contention is, given the degree of transparency we're going to need, and I might say the delicacy of the transition, you might say, in other words, getting to an international coordination that isn't a global dictator, human or artificial, and preserving democracy, at least in the sense of, you know, various nations get a say, right? I don't mean, in fact, an important part of my foreign policy ideology is we can't afford to demand that China become a democracy. I'm not sure if that's what Dario is demanding, but maybe we'll get into Dario's China view eventually. I'm just giving you a sense for the contours of my overall argument that we're going to have to move toward true global community. I'm not saying it's likely. I'm just saying that the alternative is like super bad. That's my claim. How do you respond to that alone?

All right. And I'll do it without using the word kumbaya too. You have no idea how many times, not just you, but because it is a dismissive reference to international cooperation. And I think it's bad for the vibe if everyone thinks it's uncool to talk about international cooperation as if it's possible. And that is the vibe, whether you mean it or not, when people use Kumbaya dismissively, it reinforces the vibe that... people don't talk about international cooperation. That's my. No, I think that's that's very fair. You're going to cause me to update my language on the podcast, which I think is my work here is done. If a bus runs over me tomorrow, I will die a happy man.

We all have our life goals. Yeah. So. So, OK, a couple of thoughts here and maybe I'll frame this a little bit in terms of how I've shifted my thinking on what things are worth working on and what things are not worth working on in in this space so um you know i i used to try really hard to to kind of turn the battleship so our our report in in 2024 um a follow-up report that we put out on ai security looking at what it would um one thing that i guess i didn't mention about the work we've done um so we spent about a year and a half doing uh nation state security for some of the biggest builds that are going up right now with Frontier Labs.

I'm super black-pilled on the ability to do that at all. I don't think that we're going to be able to get to the point where we have data centers that can successfully prevent adversaries from stealing critical IP or sabotaging these sites with any degree of sort of nation-state level robustness. And so, but as we put out these reports, What we're trying to do is shape public opinion and opinion in Congress at the executive level. And I'm not saying that they had no effect. There definitely were people who were at least nudged based on these things. But over and over again, what we've seen, especially in the last few months, last year maybe,

kind of Chernobyl incident will be a function of US models having kind of gone rogue or been weaponized in some interesting way. The problem is obviously you can't do this without engaging

China because they're six months behind. And so you'll just play the whole thing out. You'll get wiped out by a Chinese AI instead of American AI. And depending on how patriotic you're feeling in the moment, this may or may not be more or less satisfying to you. And so the president calls China. He calls Xi Jinping. I'll spare you the details as to why it has to be Trump to Xi, but that is not a naive view. It's literally, if you talk to State Department diplomats who've done the negotiating on arms control with China, it literally has to rise to that level to be anything serious and tractable. The president goes, hey, guess what? This is no longer tenable for. It is no longer tenable for us that you are building AI data centers at this scale, that you're doing training runs at this scale, whether you believe us or not. And this is a key piece of uncertainty that I think a lot of folks in the AI safety community do not grapple with. There is deep epistemic uncertainty that we must have about whether Xi Jinping believes any of the things that he says. He's known for charm offensives. The Chinese, the Russians historically have used this as a really effective tool, leveraging the Western polity against its own

interests. This is a really easy, it's just like the standard trick. And so we have to assume that she actually is fairly unmoved by whatever indications we have of catastrophic risk, even if there is a Chernobyl event. This doesn't mean that he doesn't believe it. I'm just saying when we're talking about the security of the United States, the security of the West broadly, we can't be resting that on the assumption that, yeah, he sees what we see, he's interpreting it the same way, and he's sincere about this. And so threats have to enter the equation. There's no other way um it is it is simply can i interrupt and say do you acknowledge that the mistrust is symmetrical and i can get back to what china would say about us and now we have not kept agreements we have violated international you know most of the most of the things we complain about there there are uh plausible i would say in some cases more plausible uh allegations uh they can make in response. So this is a symmetrical, the trust problem is symmetrical. And I want to get back to that eventually because China does have a long list of grievances. And by the way, they would say there are specific cases where contrary to stereotype, they have tried to cooperate

internationally, and we have basically betrayed them. They would point, for example, to them letting, you know, they could have vetoed the Libya intervention authorization of the Security Council. They didn't. We said it was it was to defend. It was for humanitarian reasons. We turned it into a regime change operation. I think they're right about that. They would say, look, on the Iran nuclear deal, we played a constructive role. Russia played a constructive role. It took a lot of work, some sacrifice. You know, we stayed on board and and then you guys bailed out of it. And so they have a long list of examples, not only where they actually do, in their view, in good faith, try to cooperate, but we fucked them over.

And, you know, leaving aside the kind of most fundamental complaint about the U.S. and international law, which is, you know, and agreements we've signed, we ratified the U.N. charter. You're not supposed to invade other countries or bomb them unless they do it first to you. since 1979, China has abided by that. We have not. We have on at least six, you know, we have, I mean, bombed or invaded countries maybe a dozen times, at least half a dozen of those have been in flagrant violation of international law. There are, you know, we say, well, China promised this and didn't do that. And it's usually some kind of informal promise. They have their versions of this. We promised we would reduce arms sales to Taiwan. We promised this, we promised that. And I guess I get the sense from you that you, I mean, if you're going to say you have to do this in an atmosphere of mistrust, so we have to have threats, I guess I'm asking you, do you acknowledge that we can expect China to have threats? And what are those threats going to be? And how is this all going to work out well? Yeah. So I guess a couple of things. First First of all, I think I think I must say that I I like the U.S. a hell of a lot more than

I like China for a number of reasons. The I'd rather I'd rather live here. But all we're talking about is will a nation state cooperate? That's the only thing on the table. So we can. Yeah. I think we don't have to talk about human rights in China. I think you'd agree. Right. Yeah. OK, cool. Yeah. So I just want to make it clear. i'm abstracting away that part of the discussion it is obvious that if we were going to live under under one regime uh my strong preference for a huge number of reasons would be the united states um uh and so so i'll leave the kind of you know and the nine dashed line too i think kind of is a counter example to like you know china has never they may not have explicitly bombed places but they sure have like been grabby with an awful lot of extraterritorial uh places yeah and i would say further they they did defy uh an international uh tribunal ruling which i don't approve of they do point to the fact that we have done that our you know in the case of nicaragua when the international court of justice said no you're not allowed to uh blockade them and and and and so on we said fuck you you don't have jurisdiction china said in the in this case you don't have

jurisdiction. And let me acknowledge something. The democracy, human rights issue could matter even to this discussion if you accept what I gather is Dario's view and is a common view, which is that it's China's aspiration to impose its system of governance on the world. And if it wins the race of AI, it will do that. I personally think there's no good evidence that that is its aspiration, if anything, it seems to me, that's kind of a projection in a way of our approach to international relations. And China is almost notable for the pragmatism of its international relations. But let's leave that aside. I mainly want to acknowledge that the question of internal governance is not entirely irrelevant. Daria would say it's relevant. I know a lot of people would say it's relevant and it's why we have to win a race to AI.

let's put that aside and let you continue. Sure. And, you know, I think how you define the sort of scope of China's ambitions itself is complicated. You know, they have a sort of ethnic-based understanding or ethnicity-based understanding of their own political remit, including the influence that they are entitled to exert on expats in the United States and elsewhere, you know, viewing them as kind of like organs of the Chinese state, rightful organs of the Chinese state in ways that would be alien to anybody with kind of Western sensibilities.

huge amounts of economic activity. It will be wildly unpalatable. It'll be unpalatable for China as much as for us. And they may not want to do it. They may not believe that we're actually on track to build the tech that we are. China's sort of engagement on this has been kind of interesting in and of itself, they've framed a lot of the rogue AI stuff as sort of vaguely hoaxy.

To be honest, I'm not clear how much they believe that either, just because there's so much confusion on this issue and they're trying to jockey for position. And I think views on this in general are very fluid. The American public's view of AI is very fluid. They may start getting benefits from it. In fact, I think it's going to happen that more and more Americans see actual practical benefits in their lives uh i i just think the whole thing's fluid and uh china could get uh you know super intelligence pilled tomorrow if they're not now and and and you and you'd kowski eyes or whatever but but yeah absolutely and and there's even this level of like so we talk to um folks who are like chinese um entities that are involved in setting up track one track 1.5 track two dialogues uh on the chinese side and or sorry sorry track 1.5 and track two by which i mean um sort of like chinese government to uh u.s private sector or academic and vice versa or just private sector academic private sector academic which is track two so the okay and the first of those is 1.5 that's right whereas track one is through the formal official diplomatic channels between two countries right um and so so we talked to some of these chinese entities and like one

One thing that's been interesting is someone came back and said, well, if America comes in and says, we want to control the compute, one unexpected reaction from the Chinese side may actually be confusion about why there's so much focus on compute as the driver of the risk, given that in China, unlike in the West, there's a weirder amount of influence that offbeat theories of AI have. So whole brain emulation, for example, has traction in China in a way that it just- Really? Yeah. It's viewed as this whack. hacky fringe thing here. Ben Gertzel, like, I don't know if you're familiar with him. Yeah. Yeah. Robin Hanson.

Yep. Yep. Yep. Absolutely. So, so Ben Gertzel has like singularity net and, and, and this sort of very, um, uh, vaguely esoteric way of thinking about like a decentralized computing to anyway, there are all these alternatives to the plain and simple scale this motherfucker up and let's make super intelligence play that, that wins out politically and technologically in the West. And so the one view, at least, is that China may just be genuinely confused about why we're focusing so much if the risk is artificial intelligence. Why on the compute layer? Which is a whole separate reason to say, well, look, we get to abstract all that away if we just go, listen, you're not going to build these sites. I don't care what it takes. You're not going to build these sites. And if you do, there's going to be a commensurate response from our side.

We're going to maintain, the crucial thing is we'll maintain strategic ambiguity about what our exact response will be, but we're going to be very clear with you about what our red lines are. If, so help me God, we see you build a cluster of compute above 500 megawatts, or if we see heat signatures consistent with a training run above this scale, that's going to be, I'm not going to tell you if it's cutting an undersea cable. I'm not going to tell you if it's a certain kind of cyber attack, but like there's going to be a response and it's going to be. Now, is this the case where we are unilaterally restraining ourselves and we're just asking for corresponding compliance on their part and threatening them if they don't give it. Correct. Yeah. Because in this world, we're forced into unilateral slowdown. Like there's just no other way. Again, it's like, and I understand why there would be skepticism about this particular, especially given the administration. Well, there's a credibility problem, you know, making the threat credible.

we're talking about a pretty big, you know, you're talking about attacking them. They have nuclear weapons. They might not believe it, mean it. But yeah, there are issues. But go ahead. And this goes back to like sort of what relationships between great powers currently look like and how it plays into this. So there's this there's this thing that's happening right now that's been happening for the last forever, really, where people are kicking each other under the table constantly. So this whole notion of sub-threshold warfare, where China and Russia in particular, like to go on Western soil, US soil in particular, and the Russians like to set fire to things, and the Chinese like to execute cyber attacks and also human intelligence operations, they're pretty flagrant, like that lead to IP theft, as everybody knows, and property damage, things like that. And then the killing of US citizens on US soil.

Now, I heard you allude to that. Who are the U.S. citizens who have been killed by Russian or Chinese agents? Quite a few members of the U.S. intelligence community. So these are. Why isn't this in the media? I mean, or maybe it is and I'm just slow. I'm not. No, you're right. It has not. So Havana syndrome was kind of like the the first public touch point of this issue. So, you know, we've talked to a lot of people in the intelligence community who will describe how they've had colleagues and families who've basically just been given brain cancer using these weapons, which are known to be Russian weapons, that, yeah, essentially induce this.

And the game there is to play on Western resolve. So the, and I'll just, I'll tell you the version of the story, according to some of the folks we talked to on the intelligence community side, who are now retired, had their most active years during the, in the pre-Berlin Wall era, and then continuing into the GWAT, the Global War on Terror. But what they saw was friends, family, you know, colleagues, you know, having their lives destroyed, being killed by these weapons. They claim that these weapons had the hallmarks of being, you know, Russian in particular. operations, that the U.S. administration, across administrations, the U.S. government denied publicly that this was happening. Their claim is, I can't verify this independently, I'm not, I'm not, you know, but their claim is that the government took the position publicly that this was unknown, spooky stuff, and that it was investigated. There was actually a report that came out claiming that there was more or less nothing to see here. And then more recent reports that have sort of been changing their tune a little bit. And that ultimately, a lot of these people were even, again, according to some of these claims, I can't verify these, but were denied

medical coverage, because these were not acknowledged to be, essentially, deaths in the line of fire, in a sense. And so there's a lot of outrage on this. But, yeah. Yeah, I mean, I'll just say, we shouldn't get too sidetracked by fantasy. And I'll just say, Last time I looked into it, I was skeptical that there was reliable evidence as opposed to the kind of anecdotal evidence that can can can involve a kind of contagion effect. But but let's just leave it. I haven't updated. I haven't looked at stuff. Don't don't try to convince me. It could be that there's a whole trove of new information that would change my mind.

But I don't want to distract you from the point you were going to make when you said there's this kicking under the table thing going on. Yeah. And this is also, I mean, so, you know, and anyone who's watched The Americans has a pretty good idea of like what this looks like. My wife. My wife has done that. I'll ask her. Yeah, it's a good show. But yeah, I mean, more or less, you will have people murdering each other and doing all kinds of stuff in the normal course of, like, espionage activities. But you'll also have greater sort of sabotage operations. And the purpose of a lot of that isn't the operation itself.

it's to prove to yourself that you have the technical capacity to reach out and touch a target ahead of need so usually the way this well and also to send a signal presumably or or is the expectation that they may not even know it happened it can depend depending on the operation but you're right that's absolutely like a test of resolve or a signal like hey we're unhappy for example with you guys went out and killed one of our guys so now we're going to kill one of your guys like we're keeping score here. But there's been a sense that for a long time, the U.S. response to a lot of these activities has been muted and softened by a lack of willingness to throw out the countermeasure, do the counterpunch. And the concern here is that that actually is paradoxically quite escalatory. The same argument that we got on the Havana syndrome thing, that essentially the government was concerned. So not responding is escalatory, your argument? Correct. In the same way that if you have a kid because you're not deterring it at that level that's right and so they have escalation dominance yeah exactly um you know if i if i don't discipline my my toddler for like you know whatever uh spraying crap on the wall or something um then you know this snowballs

over time if i take a completely hands-off approach i could end up with a teenager who you know does god knows what right so it's sort of the the same thing like you're teaching your adversary how to But at the same time, if both sides take the view that you do have to retaliate to deter, that leads to actual escalation. Yeah, well, um, endlessly, endlessly in principle, it can, it's a positive feedback. Well, it actually can not be. And this is actually what makes it sub threshold warfare. So the, the, the fact that you're kicking each other on there under the table, um, requires everyone to have some idea of what is escalatory and what isn't and everyone has different interpretations of that right there's you know the russians cut an undersea cable so what do we do we blow up some munitions factory is that escalatory it's asymmetric is it escalatory or is it not it's really hard to know and so that's that's one of the sources of instability here but everyone does have pretty like pretty good.

clarity on what is nuclear escalatory and what's not. And that is keeping a check on an awful lot of this behavior. So it gives you kind of a constant measure of the pulse of how your adversary is thinking about issues like that. Okay. Now, does this, the kicking on the table thing, is that relevant to how we would confer credibility on our threat to bomb their data centers or whatever? Is that? Yeah. So how to spell that out? Yeah, good. So in a couple of ways. First, by the way, it's really destabilizing because now we can have cyber attacks that we can blame and they can blame on rogue AI systems that we never actually intended to launch.

No, I emphasize this when the I guess when the hugging face thing happened on on on Twitter. This is a threat of AI that there's a kind of attack where it's hard to attribute the source. so you could have some some you know two-bit third-party actor launch a cyber attack that leads to nuclear war between us and china because of course immediately there will be a chorus of with all due respect china hawks not you but you know the kind who would say oh this must be china who did this thing and and and and so we retaliate with kinetic and it gets more kinetic and so on so anyway that's that's my sermon go ahead yeah no the kinds of schmucks would say uh kumbaya or whatever. Yeah. You know, you've met them. You've met these people. I've heard of them. But yeah, report to me when you see them. Yeah. Yeah, I will. But yes. So so this is one example or case in which it's destabilizing. But what you asked about was in what way is this the subthreshold warfare idea kicking each other under the table important for for the sort of keeping in check of your adversary in this context.

And so one key thing that it does is it proves to you and to your adversary that, again, you can reach out and touch a target. So the way we do this in practice is if we have a target that we're interested in in, say, like mainland China, we'll not go after that target per se. We'll go after a proxy for that target that is built. So I'll just kind of do a cartoonish version of this. But so let's say that there's like a certain kind of a vessel that we're concerned about the Chinese having a merchant ship, but like, you know, that may be used to do something that we're concerned about. We may look at like, okay, well, what are some ports that see ships like that, but that are owned by another country, you know, maybe out in Iran or something like that.

Let's go after that, convince ourselves we can actually execute the operation so that we know that at game time, we can actually pull the trigger and this will work. And the Chinese will see that, or they won't, depending on the purpose of the operation. But it's some combination of proving to ourselves we can do it and showing the adversary that we can. Sometimes you want to learn without teaching, and sometimes you want to learn and teach and convey a message. And so being able to show China that, like, hey, we have the ability to reach out and touch this thing. Look at this railroad. Look at this factory.

We're able to actually make SMIC suddenly grind to a halt. Um, those sorts of activities are constructive because they demonstrate to the adversary the pain could be real if you, if you start to mess around, um, as long as China does not retaliate in kind, well, in which case we may feel compelled to kick harder under the table or over the table. But it gets back to my general point about how, you know, there's a separate risk of escalation from responding to these. There is one of the stabilizing aspects though, is that the Chinese, the Russians, the America, everybody appreciates why these activities are being undertaken so you get kicked in the shins and you go ow fuck you also okay i see what you did there and that recognition is stabilizing so so like you kind of understand the chinese to be probing you so that you know and then you understand that they understand that you are probing them so that they know and every Nobody fears nuclear escalation as a baseline, and no one is intent on that.

But having this sort of steady signal, because the alternative is you find yourself in a conflict, and for the very first total crapshoot, you're testing out all of your concepts of operations, all of your new weapon systems, all of your capabilities. You're testing out also, crucially, your willingness to authorize operations. It's something that I think a lot of people kind of discount. out. So the chain of command is a chain of many people who have to say yes. We have obviously examples of like heroes in Russian subs who refuse to launch nukes when they absolutely should not have launched nukes and they probably saved the world or at least like large swaths of it. There is a flip side of this, which is the right people have to say yes in order to authorize an operation in a pinch. And you want to flex that muscle a little bit to grease the wheels so that when crunch time comes, you don't suddenly have the reflex to not do something out of fear of not being able to successfully execute the operation or unfounded fears, uncalibrated fears of escalatory risk. And so this is the thing is like escalation is a two player game and it's possible to be too escalatory. It's also possible to be under aggressive and to

teach the adversary to do more. And what we absolutely must maintain is our ability to impose consequence. If the Chinese do decide to renege on a commitment, or if they just decide to ignore our threats and warnings and go ahead with their training runs, we have to be in a position where we have and are believed to have the willingness and the capability to exercise whatever operation. Okay. So here's a question. Do you make the threat publicly? Because that if you do it publicly, I mean, it's kind of hard to imagine doing it not publicly because if we impose a moratorium on our big training runs, the president who does that is going to have a lot of explaining to do. And if he can't say, but wait, we're going to get China to do it, he's in political trouble. But if he does it publicly, I think that makes it less likely that China will buckle because it's a kind of humiliation, right?

It all depends on. So there are degrees of freedom here that are quite helpful diplomatically. And and and this is it's maybe worth getting into into those specifically. But you don't necessarily need to frame it as force and capitulation. There are options that you can put on the table to make for a more graceful. Yeah. So you can say we're optimistic that China will see the wisdom. And if they don't, we'll go back to training. And then under the table, you kick them. Okay. So let me, I mean, first of all, I want to, I mean, we can get, well, yeah, I actually have a question that responds more directly to what you're saying. But I do want to go ahead and kind of preemptively clarify a couple of things I'm saying. When you say when finally Trump crosses the threshold or the American president crosses the threshold where they see that some kind of very significant coordination between the US and China is required, and I would say different presidents have different thresholds. Let's don't get into Trump, but one can imagine a president who would be close to that threshold now. Trump, I think, is not that. But whichever president does that, I agree they're not going to say, hey, can we be friends? They're just going to say, hey, we have this

common interest. And then they may or may not kick them under the table. We can get in other

to it there that, that I think is deeply counterproductive. Um, but yeah, so I, maybe you should, maybe you should pitch a startup to Saks and try to get funding, uh, that, that does verification. But yeah, um, look, can, can I, uh, ask you a, um, so back to, uh, you know, threat

oh yeah so and actually i should clarify where like so why my emphasis is where it is um and and this might i i don't know that there's as much daylight between our positions then as it seems so um one key piece is look at the the relative postures of the us and china on this issue a lot of people tend to think oh america is like the world's preeminent superpower and therefore we're right and tight we're good to go um the chinese lab i'm sorry the um uh U.S. frontier labs. If you talk to folks in those labs, ask them, say, on the, you know, well, okay, I won't mention the team, but like on a team at OpenAI, you ask what fraction of people who know the most sensitive algorithmic secrets of this group?

are chinese nationals with by the way therefore obligations to the chinese state it's just a simple matter of fact well in china's view oh right but but these okay so with with uh exquisite levels of leverage that the chinese state is able to bring to bear on these individuals right we're talking denying their grandmother or their mother insulin medication preventing their brother from traveling or opening a business are there documented cases of things that extreme the the insulin i mean is that on the record um or you it's it's on the record because of a report that we put out about 18 months ago where we talked to people from the intelligence community who were describing a case that was not yet public. Actually, so this example is kind of informative.

So there was. And are there names associated? We can go talk to the we can we know who the person is whose relative was threatened. And that's on the record. No, no. So, no, that's a good point. So, yeah, in a sense, you have only my word on this. Yeah. So that's actually an important update. you know calibrate accordingly um but uh the the story that we were told was or a story was that there was a power outage at oakland that affected berkeley and that um so the the dorm power went out and all these like chinese nationals on the floor were freaking out because they had an obligation to do a time-based check-in with their handlers in beijing through the consulate or through the you know the diplomatic office and uh so so the we've had this described to us on a number of occasions by folks at the FBI, at the DIU, and so on, who've had to kind of map these networks.

And like China's ability, so they have a bunch of different consulates, a bunch of different diplomatic outposts. They all manage their people in different ways. But Chinese nationals have an obligation to report to these outposts. And those obligations are enforced. Like, you know, there's obviously a social credit system in China that actually materially affects your life prospects and so on like the ability and and level of granularity with which they can put the squeeze on somebody is pretty extreme and so when you look at the frontier labs that are recruiting from from berkeley not just from berkeley explicitly from xinghua university from the beijing academy of ai from all these like explicitly people as like pla affiliated uh organizations um like it's it's a lot like there are literally dozens of open ai employees that come from you know xinghua or buy or these sorts of institutions, and even more Chinese nationals who've been trained at institutions like Berkeley. And so when you ask some of the people in these groups, like what percentage have access to these algorithmic secrets, they'll tell you like, it's not double digit percentages, it's like sometimes north of 50%. And so the placement and access that China

enjoys, and by the way, this is also true of like all layers of the stack. It's extra true at the frontier lab level, partly because of the naivete of like Silicon Valley people who for years were just like oh another fun anecdote um so uh a guy we know was uh he sort of playfully asked me he's like so um let's talk about russia for a minute of the top 10 um known and prioritized operatives russian operatives working in the united states um in uh this was circa 2020 sorry 2014 he asks me what um what fraction would you guess were uh sorry he says where where do you think they were And I go, well, D.C. is the spy capital of the world.

So, of course, you're probably a good fraction in D.C. And he goes, oh, no, none of them were there. 100% of the top 10 were in Silicon Valley. And he asks me, since then, do you think the tech universe has become more or less interesting from a national security standpoint? So the labs are this, like, prioritized target for collection and for sabotage and all the other fun things you could imagine. So you're deploying this in the service of explaining a kind of threat China could make or a kind of transparency they have, a kind of insight they have? Yes. At all layers of the stack, so we're going to come out with some investigations on this stuff.

This is true at the level of the data layer. If you look at the movements, who's actually selling data to whom and how are Frontier Labs consuming that data, there's a Chinese footprint there. If you look at it, like a big one, I'm not just saying like a one-off thing. I mean like things that matter at scale, even for the behaviors of these models. It applies at the data center layer. So if you look at like who's building data centers in the United States, often on suspiciously short time horizons, being able to get access to transformers in six months when they should be on three-year back order from China. Like this is like their level of penetration of the North American tech stack is obscene in a way that it just isn't reciprocally. And so from a human placement standpoint, from a supply chain access standpoint, China just enjoys advantages that we do not have symmetrically. And we have a lot of capability that we need to build up to be able to get to parity on this stuff. Our one last best hope right now, the big kind of equalizer is cyber. And the problem with cyber is increasingly there's this view and we're skeptical of aspects. So we, in other words, thanks perhaps to mythos, have superior cyber

capability, you're saying? Right. Well, that's right. So nominally, so the CIA and the NSA have really doubled and tripled down on cyber for a whole bunch of reasons, some of which have to do with risk appetite and not wanting to put people out in the field as much um but also just like the leverage that comes from technology um so they're they're really over leveraged on cyber right now and physical placement and access and supply chain like these sorts of things that we need to kind of but it's it's a problematic kind of leverage yeah exactly exactly like china is in our bones in a way that we are not in their bones and in terms of the the collection ability right so the ability to to verify in informal ways um that you know hey is is open ai letting a rip secretly and like running a big well let's ask any one of our like you know dozens and dozens of people who are in the lab some of which are explicitly spies but most of whom are just people we have leverage over let's just ask them like that's one option you have you also have espionage in terms of physical implants in the data centers supply chain attacks all these things they can do that are much harder i'm not saying we don't do them but they're a lot harder for us to do

for all kinds of reasons over in China. And so we need to get to parity. Cyber right now is a really important lever that we have that we're relying on too much. And I say too much because it's entirely plausible that as AI systems get better and better, they get better at defense than at offense. So like a given piece of- That's not a consensus view, right? But it is your view. So we have another report that we'll be putting out on this. It has been a semi-consensus view at Anthropic, among some of their security folks for a little while now, just as they've seen trends. So the argument is something like this, like a given piece of software will have finite complexity.

There are only so many ways that you can hack it or crack it. So if you direct a sufficiently powerful model or agent at that piece of software, eventually you'll buff out basically all the vulnerabilities that actually do exist. Now, this is fallible. I think it's a weaker argument than it seems in a whole bunch of ways that don't really matter for this discussion. But the point is that you actually could enter a regime in which you actually do have pretty surprisingly good cyber defense versus cyber offense, where the kinds of vulnerabilities that we find and look for get a lot harder to find and look for in the kind of post-AI world of like just 18 months from now. If you do that, suddenly offensive cyber operations do not give us leverage that we're used to. And we find ourselves in a world where from a cyber standpoint, we've got nothing too special. China, though, has supply chain, humans. They've got physical access, all kinds of things that we don't have. And so the ability to actually enforce, if China then goes to try to renege on their commitments, suddenly vanishes and is seen by China.

vanish, which emboldens them to do more. And so there's a sense in which the lack of parity, this is one of my big issues with Dan Hendricks's main framework, is I don't think it accounts for the objective situation on the ground as seen by a lot of folks. I'm not aware of like, you know, special like saps, like special access programs, dark black programs or whatever that I'm not aware of, obviously. But by and large, from what I've seen and heard, it seems like this is more or less the situation. And so I'm much more concerned about U.S. offensive parity than And I am about China's ability to do things that would knock, you know, our stuff out of the swim, say, you know, anyway, that seems to me to be like a really important element.

So I know you've got to go in a few minutes and I have, there's two questions I'm dying to ask you. So I'm afraid we're going to have to cut to those. One is, I'm sure you've read the part of Dario's Machines of Loving Grace paper where he talks about the need for a block of democracies to beat a block of authoritarian countries, obviously led by China to superintelligence. intelligence use that to i think it's fair to say he wants to bring them to their knees but in any event yeah what he wants him to and and demand his demand as i recall is i think there's almost verbatim is to get china to quote stop competing with democracies now first of all i don't know exactly what he means by that i've never heard anyone ask him some people think he means regime change in china some people think he means something else so my first question is do you have a sense for what he means? And secondly, what is your view on this overall question? Do you share what seems to be one of his assumptions that China does want to impose a system of governance on us? And for that reason alone, it's the race to superintelligence, if we can't avoid it, is existential. Of course, in your view, it's existential either way, because we're toast.

we're toast if if the planet gets there but uh if you could tell me what you think his view is and then what your related view is on china's kind of foreign policy aspiration yeah it was so i mean my answer in a weird way comes from the same the same uncertainty in both cases i i i uh i don't know what dario i so i've transparent like i've met dario i talked to him one time um like two or three years ago or something right so it was like an hour-long conversation and that's about I don't know where his head is at and, you know, the specific meaning of the words. I generally, the threat model I've just laid out here is certainly one in which you don't need to frame things in quite the way that Dario has. It's in the self-interest of the Chinese and the Americans to do this right. But that self-interest is, in my view, insufficient to kind of cause everybody to hold to their ends of the bargain here.

America needs more offensive heft and the willingness to deploy that. And yes, it needs verification tech and so on. I think, you know, like a block of aligned powers is actually really important for the West from a supply chain standpoint. It gives us just more leverage. I would rather the West end up with much more leverage than the Chinese. I'd rather approximately the situation be reversed. You know, right now, one thing we didn't talk about is robotics and the massive, massive advantage that the chinese have if you just let things play out status quo right now we are fucked we are so fucked i mean if you look at the the dark factories that the chinese are spinning up the scale at which they're just pumping out robots humanoid robots like everybody talks about the aging chinese population forget it not going to be an issue like this is just uh you know it may be an issue for them domestically in terms of you know the the jobs issue may be an issue for both but go ahead if they can subsidize you know universal basic income or like true total luxury communism or whatever through through this sort of thing yeah i mean anyway so um in in terms of uh whether yeah whether need that block i think the scenario as i see it is uh you need first and foremost verification

from that first moment of course verification using like you have satellite telemetry and various other collection options um that the the national security apparatus already has vetted and available to it and that it institutionally knows how to process and trust um you you're going to kind of start to back into more and more fine-grained tech and and that is more and more permissive as get more and more resolution and more and more access and as we do trust building i will say um historically the chinese response when when american diplomats say we need trust building measures the chinese response is to say well we can't do that because we don't trust you like this is like literally how but they but they they believe that and we feel the same way about them right but like the trust building measures the purpose of them is explicitly to get us to that next level of trust. So there is this sense in which the adversarial dynamic, like at least in ways that are intelligible to us, and I think to your point, this is important, we are interpreting it as meaning that they are not engaging in good faith in that way. It is possible there's a story and reasonably likely, of course, that there's a story they're telling themselves about how

the situation is quite the opposite. But yeah, so block of aligned countries, I think is really good insofar as it gives us leverage. By the way, I think non-kinetic, non-cyber options are really the first thing we need to look for. There are ways, when you talk to some of the supply chain people, of doing one-shot, six-month slowdowns of China's, for example, chip ecosystem, where, you know, you only can use it once, but you can use that card for a six-month slowdown. So the kind of thing that could be fairly punitive and reasonably non-escalatory, but you know you want that wider range of optionality um it helps you across that spectrum to have a block of allied countries and i think certainly more engagement with the five eyes um and nato uh would not be a bad thing for the u.s at this point and and you agree that the alignment should be explicitly ideological in other words you know dario's view these are democracies so i guess if even if russia just begged to be on board and they whatever we'd say, no, sorry, your elections are phony. I personally think there's real danger in casting the narrative that way. I think it tends to become a self-fulfilling prophecy of a bipolar, you know, ideologically opposed world. But is it your view that ideology should be a big part of

the litmus test for inclusion? I'm fairly ambivalent, honestly, probably because I haven't thought enough about it. I mean, you know, like anyone, my reflex would be I want the democracies to win um but with zooming out you know there's just no winning to be had in a super intelligence loss of control race and so i think it's really just about trust like and and not even about trust but our ability to verify so if we have a country like france where as you said like our friendliness to france just means uh that we have all kinds of ways of monitoring what's going on there that we don't have with russia yeah to me that's just an extra tick in the verification box for France. And it happens to be that a lot of the countries for which we enjoy that level of closeness, we have that relationship because they are pluralistic liberal democracies and all those wonderful things. But as a counterpoint to, I guess, Dario's framing, it's not obvious to me that Russia gleefully jumps into bed with China on this one. North Korea is at a fraught relationship with the Chinese and the Russians. These are not coherent ideological blocks. I mean, socialism with Chinese characteristics, like the second part of that phrase is not social, is not socialism.

Right. And just as Russia is. Well, in Russia, I mean, look, I wish people would quit calling China a communist party. It is run by, I mean, a communist country. It is run by a party called the communist party. But anyway, go ahead. Yeah, exactly. And in much the same way as, you know, Russia is a sort of like a vague. really fascist-flavored authoritarianism with religious overtones. I don't know what term is right for these things. And the beauty is, on my worldview, I don't particularly have to care. What I care about is, don't build the fucking superintelligence that gets us killed. Let's have the verification tech in place, and let's have the consequence. And the block here is useful to the extent that it helps us impose consequence for defection. So having, for example, Korea on board, South Korea, to be clear, is important because Samsung is there, because TSMC is, because SK Hynix is there, right? Having Taiwan on board is important because of TSMC.

So like, these are all helpful things. We expect to be able to monitor what they're up to so that our concern will be mirrored hopefully by them. But like, you know, beyond that, like, let's just make sure we don't do the stupid thing. And so I think that this is one sense in which, you know, I may be wrong in all kinds of fun and interesting ways, but this worldview is maybe a little bit more robust than the one dario uh is kind of putting forward okay well uh we end on a note agreement dario's view is not necessarily the the optimal uh guiding light uh but smart guy i think he's sincere i think he's ideologically sincere i do not have the degree of cynicism that i i bring to bear on uh sam ultimately agree which isn't to say which isn't to say i want to appoint Dario uh you know make him yeah in charge of world affairs I wouldn't want to appoint myself yeah speak for yourself I would want to appoint myself but I I'm starting to think I should give up on that career aspiration and you're young don't give up on it you know I just have great skin yeah well I I wish I did uh or we're young um so anyway thanks it's been a great conversation that's where we can talk about uh i so i hope you'll come back at some point and we can um

keep yeah same here going and uh again uh people should check out the podcast and all your uh all your reports uh of uh gladstone ai that is uh that's the name right yeah ai.com for the newsletter if that. Okay. Right. Exactly. Is that on Substack? Yeah. Okay. Good. So check it out, folks. And we'll see you next time, Jeremy. Thank you.