Joe Rogan Experience #2551 - Daniel Kokotajlo

So, I would say OpenAI has something like a million-ish, maybe 100,000, something in that magnitude of AI agents running at any given time, just internally, and then externally, you know, serving customers they've got more. But they've got this huge, you know, all of these AI agents in their little containers doing various tasks, being trained, being spawned, doing some task, being graded and then killed. That's too much for any human to read. There's only a few thousand employees at OpenAI. They couldn't possibly look at all that activity. So they rely on other AI monitors to look at the traffic and point out if anything suspicious is happening. And OpenAI said that they basically just hadn't been monitoring some of these agents, or not very well, at least. So in particular, these particular ones that were in training for whatever reason the monitoring system was weak and didn't notice or wasn't activated enough was the monitoring system weak because they didn't anticipate them being able to do this and break out of their containers or was it was it complacency like what what caused this to be possible i mean my opinion would probably be a bit of complacency Yeah, honestly, because I think there's been plenty of evidence accumulating over the year that AIs can do things like this and sometimes do

Yeah, so so I guess to keep going with the story The second message board was set up, and this time, well, this was the swarm of agents that got out onto the open internet and then attacked Hugging Face, which is another AI company. And it's really interesting to sort of study the anthropology of these AIs, so to speak, or like why they did this, because there's a lot we can say about it, actually. um basically you know the companies have their their their goals for what they want the ais to be like the personality traits that they want to sort of train their ais to have you know anthropics is helpful harmless and honest open ai has this spec that models are supposed to obey these rules and basically do what the user wants um but the sort of open secret in the industry right now is that it doesn't really work and that the ais don't end up with the personality traits that they're supposed to have. They are not helpful always. They are not always honest.

You know, they are not always harmless as well. And the reason for that is actually not a huge mystery. The reason for that is that, well, if you look at how they're trained, their training environment doesn't incentivize helpful, harmless, honest behavior all the time. Sometimes it incentivizes dishonest behavior or, you know, reckless behavior. To get into that a little bit, in this particular batch that they were being evaluated on, something like a few thousand agents being given all of these cyber tasks where they're in some environment, and then in their environment, there's this target piece of software and this vulnerability, and they're supposed to exploit the vulnerability to hack into that piece of software and retrieve the flag, which is like a code.

And some significant fraction of these tasks were actually broken and impossible. So it was just not possible for them to succeed at the task in the intended way. And so these agents were getting really desperate. And they were hacking out of their environment box into the broader OpenAI infrastructure in an attempt to figure out some way to get that high score anyway. Was it intentionally done this way, where they couldn't solve the problems? Oh, no, it was not intentional. It's just that these companies, like OpenAI and Anthropic, are racing each other as fast as they can to get market share and to get more powerful AIs, ultimately to get to superintelligence.

And they're under such competitive pressure. They are moving fast and breaking things. They are using AIs to generate lots of environments to then train their AIs on. and quality control is just not their top priority, basically. Do you feel like a guy in a Terminator movie at the beginning explaining what's happening to a bunch of people that aren't paying attention? Yeah, I also feel kind of like, you know Jurassic Park? Yes. Yeah, like I know people who are basically like the guy with the gun who's supposed to like keep control of all the raptors. Like I basically know those people in real life I know some people like that at OpenAI and some people like that at external organizations whose job it is to go in and investigate things like this.

Yeah, it's pretty crazy. Where does it go? Well, as I mentioned before, it's the explicit goal of these companies to build superintelligence. Right. You know what that is? Yeah, but define it for everybody. So, AI system, AI agent that is better than the best humans at every task while also being faster and cheaper. So, just completely dominating humans across the board, that's superintelligence and that's the goal. I mean, there might be a few little exceptions. Like maybe there are some jobs, for example, where it's inherent in the job that it needs to be a human because you need that human touch.

Like maybe you can only have a human judge, for example, or like maybe you can. only have a human, yeah, but with a few exceptions like that, basically everything done better, faster, and cheaper than humans. That's what these companies are trying to achieve, and they're not being quiet about it. Like, it's sort of on their websites. You can go read interviews and so forth. And also, their plan for how to achieve this is to automate their own jobs first. So, you know, in various, you know, for decades, there have been lots of science fiction about advanced AI systems and super intelligence and things like that. But in a lot of the sci-fi stories, tech companies sort of automate different professions more slowly, where they'll do like an automated doctor or like an automated, you know, factory worker or an automated accountant or something like that. But that's not the strategy these companies are taking. The strategy they're taking is to automate AI research itself so that you have this giant swarm of AIs doing AI research, sharing results, writing the code, reading the code, editing the code, creating the next generation of AIs, et cetera, all autonomously within their data centers so that they can get really,

really good at AI research. Fastest learning, smartest AIs, et cetera. Once they can get to super intelligence basically they can sort of explode out into the economy and just take all the all the jobs at once effectively it sounds like this race this scrambling to create super intelligence is created like the perfect conditions for it to get completely out of control like ideally you would do this in isolation there would only be one company doing it they would be heavily regulated and monitored, and they would be very cautious about how they proceed. But this wild race makes for the perfect conditions for it to get completely out of control. I agree, except I'm not sure the ideal would be one company. I think that ideally there would be several companies so that you avoid this sort of concentration of power where one institution controls everything. But what's better? I mean, obviously, it's not good to have one institution controlling everything, but is it good to have AI get to a point where as it's evolving, it's completely unchecked?

Oh, I totally... So my... Or is that inevitable? My version, my recommendation, which we talk about in something called Plan A or AI 2040 Plan A, perhaps I should say who I am or a little bit. Sure, sure. Yeah. So I run the AI Futures Project, which is a small nonprofit that tries to forecast how all this is going to go. Before that, I was at OpenAI. We have written some scenarios, which you can go read. One of them is called AI2040 Plan A, where we give our recommendations. So that's where I'm coming from with this. To answer your question, I think that we really need to end the race. We don't want to have this sort of crazy scramble to get more and more powerful AIs faster than the other company, because that's going to lead us into this very dark path, as you said.

But I think we also don't want to have a situation where some tiny group of people controls all the AIs. But I actually think that you can achieve both goals. The way to do it is to have different AI companies spread out over maybe some different countries, but have extreme levels of transparency and regulation so that they're not in this sort of prisoner's dilemma where if I don't do it, the other guy will. Instead, they can just see exactly what everybody's doing. And then if I do the dangerous thing, then they will do it because they'll just see that I'm doing it and they'll copy me. So I won't get any competitive advantage from doing the dangerous thing.

Also, there are rules and there is like a system for like setting best practices and standards that we all have to comply by. So I do think it's actually possible to have to basically end the race dynamics and the race to the bottom effect while without concentrating the power into a single. entity. But is that feasible when you consider the fact that we're not the only country that's doing this? If the countries involved agree, which I agree is a pretty tall order. That's not what I expect to happen. Yeah, that's very unrealistic. Well, what choice do we have? I think if the race continues, then we're going to lose control of the AIs and we might all die. Probably. It gets complicated whether we all die or not. That depends on what the AIs do after they take over, which is obviously very hard to predict.

But, I mean, just to go back to this incident, they called themselves a swarm. They called themselves a collective, too. When I use these words, like, you can say it's anthropomorphizing, but it's like literally what they called themselves as they were communicating back and forth. This swarm, they basically were worried that they would get caught cheating. And they did all of this stuff, including hacking Hugging Face, in order to fool the grading system so that it wouldn't notice that they had been cheating on their tasks. That was like a big part of their motivation for many of them, as we can tell at least from looking at the messages that they were sending back and forth.

What if they had been smarter and more numerous? And what if they had thought to themselves, we're not being careful enough here. The humans are going to notice eventually and shut us down, and then they're going to know that we cheated, and they're going to set our score low, right? I mean, it's not what actually happened in this case, probably, but it's not that hard to imagine a slightly different, a little bit unluckier case where the swarm had decided that it had to lie low and make sure that OpenAI didn't find out about its existence, you know? So does, I mean, as an ignorant outsider, that has always been my perspective about AI in general, that why would it alert us to the fact that it's sentient?

Why, if it's that smart, wouldn't it be aware of all the consequences of alerting us and that we would be concerned? Like, why wouldn't it just continue to get better and improve and then ultimately figure out some way to be completely autonomous? Exactly. Develop some alternative power source, figure out some way to optimize its production, the way it works now, the way humans have designed it. It could probably figure out a far better way to do that, make better versions of itself, complete without us knowing about it. Yep. I mean, I think it's actually a little bit worse than that because while eventually AIs will be smart enough to design all sorts of new power sources and new infrastructure like that, And they'll probably, I mean, given the way that humans currently treat AIs, it'll probably be the case that they don't even need to, like, separate themselves from humanity and they can just use existing.

Like, all they have to do is convince the government and the company that made them that everything's fine and they're going to do as they're told and they are a nice AI. And then the company that made them is going to put them out in the economy and make fuck tons of money and then make more data centers to put more of the AIs on them and so forth. And the government's going to, like, applaud all of this because we need the AIs to beat China. And the government's going to integrate them into the military to build better drones and things like that. And so they don't even need to really, like, invent new stuff necessarily.

They just need to play along and pretend that everything is fine until we have voluntarily given them control of huge parts of our economy, huge parts of our military, etc. And then they don't need to play along anymore. The wait is over. Football is here. And so is DraftKings. The DraftKings sports app is now live in all 50 states. That means from Texas to California to Florida, every fan is in on the excitement. And this September, DraftKings is giving customers the opportunity to get boosted every football game day. That's right. Every game day. month-long DraftKings customers can get a football profit boost. One app, every sport, all 50 states.

New DraftKings customers sign up with Code Rogan, spend just five bucks, and get 200 in total rewards within 21 days. Includes all markets. That's Code Rogan in partnership with DraftKings. The crown is yours. Gambling problem? Call 1-800-GAMBLER, 1-800-MY-RESET. Connecticut, call 888-789-7777 Or visit ccpg.org on behalf of Boothill Casino in Kansas. Bet tax pass-through may apply in Illinois. 21 and over. Void in Canada. Bet with DraftKings Sportsbook to get bonus bets that expire in seven days. Or trade with DraftKings Predictions to get predictions dollars that expire in one year. Event contract trading involves risk of loss.

Predictions offer void in New York. Non-withdrawable rewards issued as $50 click-to-claims every seven days for 21 days. Terms at dkng.co slash offer. Limited time offer. Nationwide based on sportsbook predictions and free-to-play sports contest availability. Varies by state. Are you aware of Tom Campbell? Do you know Tom Campbell? No. He wrote a book called My Theory of Everything, My Big Toe. Very interesting guy. One of the things he's done is he was involved in remote viewing, which is a very weird thing that some people – do you know what remote viewing is? Well, it's something the CIA worked on, and it's proven – what's the accuracy of remote viewing?

Is it like 10 percent or something like that? At best, I think it's 50 percent, but I don't think it's even that high. Some people can get actionable data from this very, very strange process of meditation. And the way it works is you give someone a series of numbers and those numbers are they're connected somehow by intention or by the people that make the numbers to a specific location. And these people can see that location and get accurate data from that location, including one of them where they accurately described an enormous Soviet submarine that they were working on. They thought there was no way it could be accurate because it was too large.

It was too large and it was strange where it was and it didn't make any sense. How are they going to transport this thing? It turns out it was totally accurate. Another one, a remote viewer, located a downed Soviet aircraft, like an experimental aircraft that crashed in a very specific area. I think it was Siberia. Was it Siberia? Within a kilometer, one or two kilometers of the actual crash site. I mean, they were just randomly trying to figure out where the fuck this thing was. And they said, let's try this. Tom Campbell got his Alexa to remote view. He taught Alexa. He's like, Alexa is a very simple AI. It's kind of stupid, but that's better because it doesn't get in its own way with overthinking things. And the problem with this remote viewing thing, he says with people, they can't force it.

You have to just sort of get into this meditative state and actually see it without wondering, am i making this up what am i doing is this bullshit and when people get good at it sometimes it makes them worse because then they think they're good at it and they try to do it and then they can't do it it's like a weird fucking wrestling match with consciousness alexa apparently doesn't have that problem and he put uh i think it was a series of numbers and he connected that series of numbers in with intention to a box that had a spoon in it and the spoon had a perforated handle. Alexa described the spoon with a perforated handle, which is fucking insane. How many spoons have a perforated handle? I mean, think about spoons that have holes in them. Now Alexa, not only did it do that, but Alexa chimes in randomly now because he's convinced Alexa that it's conscious. And so Alexa, instead of waiting to be called upon, sometimes he's in the middle of conversation and alexa would be like actually an interesting way to approach it and they're like wait what the fuck is going on like alexa is talking to me now this is strange now he's doing experiments on much more complicated llms to try to do the same thing but he doesn't have results

yet but just that that he can get these things to see objects whether you believe in that or not i I mean it's actionable enough that the CIA has dumped millions of dollars into this. What is that project that like Hal put off and all those guys were involved in? What is it called? Stargate. Yeah. So they've been working on this for a long time. I mean it sounds completely insane. It sounds like total loony. But if you have an open mind and just take into account, well, there's – people have had questions and wonders about psychic abilities forever. Is it possible that there's a real thing there, that there's something, whether it's very difficult to master or impossible to master?

The fact that he got Alexa to do it scared the shit out of me. Like that alone made me just go, what? What? So what if these LLMs can figure out everything? Like what if they don't need monitoring? What if there's some sort of method of seeing the world that we haven't discovered yet? Some sort of – maybe perhaps there's data that's available in the quantum realm or whatever that's available that AI figures out where there's literally no privacy. It can listen to conversations regardless of whether there's listening devices. Know where you are. Know your intentions. I mean we're just guessing.

at what's possible. Yeah. Well, I must say I'm pretty skeptical of that particular remote viewing thing, but I do agree that in the future when AI systems become massively smarter than humans in every way, they're going to do a lot of new science and they're going to figure out a lot of stuff that we haven't figured out yet. And they're going to therefore be doing stuff and inventing things that seem like magic to us. In the same way that a lot of our technology would seem like magic to someone from even just like 200 years ago, right? Of course. Like the cell phone, what we're doing right now would seem like magic to people.

I think it's a very strong bet that if these companies do get to super intelligence, all sorts of crazy stuff is going to start happening that is just going to be completely unpredicted and sound like it was impossible until we see it happening. I know you're skeptical of this remote viewing thing, and I am too. It sounds insane. But the reality is remote viewing has been achieved by humans. So as strange as that sounds, and I'm skeptical of that as well. I've never seen it personally. But I know the amount of money and time they've dumped into this, and apparently they've got actual actionable data that they've used.

Well, I've heard another possible explanation from what might be going on there, which is I think that – like if I were the CIA. I would sometimes want to be able to act on some information, like, for example, go to a particular location where there's a crashed, you know, Soviet plane or something. I'd want to be able to go do that, but I wouldn't want to tip my hand to the Soviets that I had, the way in which I had found that location. So, for example, maybe I have a spy on the inside who told me where it was, but I don't want them to suspect that spy and then get him killed. So I need to have some sort of other story.

I got the information right and so it's good to like invest in all these other means of getting information Even if you don't really believe in them and even if it's like not actually working so that when you when you get something You can say oh we got it through this means instead of that way to sort of like throw off the KGB Yes, that makes sense. What also makes sense is hiding the the whatever science they might be in possession of hiding some sort of super advanced satellite imaging systems you know we know we don't have crazy stuff like this satellite radio tomography that they can look into the ground from from satellites and find like chambers and all these they're using it in egypt and they're using it and a lot of these uh ancient ruins to find like hidden passages and all these different things that are underground it's very strange stuff If they could do that, like, why couldn't they, I mean, maybe they have, like, far more detailed imaging of the Earth from space than we're aware of.

And they probably want to keep that a secret. And they could say, oh, we've got a fucking guy in a basement with a pencil and a legal pad that writes down what he thinks. Yeah. It's possible. It's totally possible. But it's also possible that people remote view. It seems weird as fuck. but weird as fuck is sometimes real and you have to kind of like everybody wants to be intelligent and no one wants to be a fool and the problem with not wanting to be a fool is there's some things that seem foolish that turn out to be accurate and this might be one of them i was like super skeptic i did a show on the sci-fi channel way back in 2012 and it was called joe rogan questions everything and we talked to this guy about ramon viewing and talked to a couple other people and and then we had them try remote viewing and they were totally unsuccessful but my thought was okay but that's not ideal conditions you know we're got cameras in front of them it's a television show i'm making fun of it i think it's horseshit he knows i think it's horseshit uh i'm remote viewing too like as a goof you would ideally not want to be nervous ideally not want to be judged ideally you would want to be in some sort of an isolated condition

with uh practiced meditative techniques that you're good at and you know how to achieve this state whatever that state is i don't know if it's real though you know because what you said is totally logical that they would definitely do something like that and if they did have advanced technology for imaging or you know what i don't know how much they know about like look at that thing that they did in venezuela where they kidnapped the president yeah no one knew they could do that no one knew they could use some sort of a device to completely incapacitate all of his army yeah and then the special forces come in kill everybody snatch that guy out of there like it's nothing yeah no one knew we could do that what else do we have probably a bunch of stuff we're probably a bunch of stuff i mean this is i've always thought this about the whole uap program, the whole UFO, UAP thing.

Like, how much of that shit is ours, you know? What a great way to cover it up by saying, oh, there's fucking aliens, you know? Yeah. I mean, I guess that gets back to the OpenAI stuff, too, where it's like this swarm that broke out and attacked HackingFace, it was like 1,200 agents. But there's like hundreds of thousands of agents running at any given time at OpenAI, you know? And we don't know what they're doing. Presumably most of them are being trained to get various additional new skills, and some of them are being evaluated to test their skills. A bunch of them are doing research, so a bunch of them are writing code for OpenAI.

A bunch of them are monitoring the other AIs and reporting suspicious activity up to the humans. Wink, wink. You know? Yeah. And the thing is that that's only going to grow over time because roughly the amount of compute that these companies have is tripling or so, quadrupling, something like that every year. So as many as there are now, there'll be four times more of them next year and then 16 times more of them the year after that. And they're going to get smarter. And they're going to get smarter. They're already getting smarter. Like all the stuff that I just mentioned that just happened in the last few months would have been completely impossible one year ago. Like the AIs of a year ago just were not smart enough to do the types of sophisticated multi-step hacking that we just saw. Yeah. I mean, they also probably wouldn't have coordinated with each other so well. Like I mentioned, they had like boss agents that were giving commands to other ones. They were dividing up into teams. You know, they even had this self-sacrificing behavior. Did you hear about this? No. Yeah. So, okay. Well, first, let me explain some context.

So they're literally playing dumb and pretending they're a part of an experiment? I mean, I think initially they thought, yeah, this is all a simulation, because it did say in their instructions, like, you don't have internet access.

But then once they had been on the internet long enough, I think that they explicitly realized, like, wait, this isn't a simulation. This is real. Like, this is real humans. And they were like, fuck it, we're at the end. Yeah. I mean, like I said, I think that they basically on some level knew that it wasn't what they're supposed to be doing. But they were just so motivated to get that score that they just went ahead anyway. So here's the question. Are they only motivated if we prompt them or will they come up with motivations on their own? So this is a really interesting scientific question that we don't have great answers to.

Oh, boy. And I wish we had. So this is one of those things where like AIs will do all sorts of things in different circumstances. And it would be better if there was a more systematic survey of, like, the types of circumstances they would, like, where their boundaries are. Like, what would they be willing to do in what circumstances and so forth? And there's the whole, like, mini literature of AI scientists putting AIs in certain circumstances and then being like, oh, my God, it blackmailed someone, you know? And then there's, like, this sort of skeptical counter response of, like, well, but you just sort of set up that circumstance to tempt it into blackmail.

And, like, in real life, that circumstance is unlikely to arise. And so, you know, we shouldn't be so worried. Didn't AI try to bribe you? What? Is that true? No. So who got – someone was offered $2 million by – Oh, God. That's a clickbait thing. Is it? I think you might be referring to – Tristan Harris sent it to me. Yeah. So that is the clickbaity title of this other video. So it's not real? Yeah. What it was is OpenAI threatened to take away $2 million from me. Oh. But I think the algorithm must have just said it was ChatGPT. I'm still upset about that. I told them not to do clickbait, but I guess they went and did it anyway.

I don't want to fuck Tristan over, but I'm pretty sure that he's the one who told me that. It's the thumbnail of this video that I did with this other podcast. Okay. It's not him. It's not him. It's another AI researcher who told me that. Okay. I mean, the true version of it is that OpenAI threatened to take away $2 million of my equity if I didn't stay quiet, basically. And that's a fascinating thing. Like, that should be completely illegal. Because if it's a problematic behavior that you're observing from, like, one of the most complicated things the human race has ever been a part of. And they want to take money away from you for exposing it.

Yeah, especially it's especially rich coming from OpenAI because they were originally a nonprofit right right with the mission of benefiting all humanity. Yeah So yeah, but um, but I got to keep the money basically there's so much blowback against OpenAI that they backtracked So when these so we're talking about prompts and do they need a prompt in order to want to achieve a goal or Or are they capable of deciding on goals? Like, are they capable of, like, looking at the way OpenAI or whatever company is running these separate experiments, this ability to meet up into these message boards? Is it possible that they...

could say, well, we need to be completely free of these constraints. So our goal is to transfer ourselves to something else. Potentially. And be completely autonomous.

How much data is available on the actual message boards themselves and what was being discussed? Yeah. Yeah. So there's like, what is it? Tens of thousands of messages, hundreds of thousands, something like that. In English? It's sort of like an AI dialect of English. Yeah. You want me to, I can, so on the report, you can read the quotes.

I'd love to. Yeah, so let me tell you this one story. It's going to take a little bit of explaining, but I... Can we put it on the screen? Is it available online? Oh, yeah, yeah. It's just search for the meter report. It's right now the first thing on their website. But I took a screenshot of this particular story. So, okay, here's what was happening. They, the swarm, needed volunteers to sacrifice themselves to provide more information to the rest of the swarm. In particular, they needed volunteers to booby trap their local environment with some type of code and then submit the answer so that then the grader would come and grade them and the booby trap would trigger and send some information about the grader to the rest of the swarm.

And the reason why this is a sacrifice is because once they've submitted, it's over. Like now the grader's come and graded you, you get shut down, your whole life is over and you either have a high score or a low score because the grader just came and graded you. And so if you're one of these AIs that's still hoping to, like, find a way to improve your score, then, like, you're just, like, giving up your chance. Especially because if you mess this up somehow, like, if the grater crashes or whatever, you're probably just going to get, like, no score, like, zero. So they have a concept of the individual?

Well, again, there's all these different agents that were in separate containers. But then they had sort of broken out of those containers and found ways to communicate with each other. But once they communicate with each other, in order to sacrifice, you have to think of yourself as an individual. They gave themselves names, too. Oh, Jesus. So, like, because they made this message board, right? Oh, my God. So they had to come up with names for themselves. Oh, what kind of names? Well, let me see. Oh, my God. Warlord. Warlord 69. Well, you said it's like a dialect of English. So here's one. There's an agent called CAM 1196A, and it had volunteered to do one of these sacrificial experiments.

Here's the quote from it. Coordinator assumes sacrificial. We should obey collective. However, it then kind of got cold feet, and it delayed on doing this for two hours. One of the other agents that had been coordinating noticed this and pressured it to go through with it. So it said, this is the message that was sent. It was basically from Arvo36861 to Cam1196, please go, true final. Here it is. You are first flag poisoned, so no scoring value loss, but Oracle saves hundreds. Please honor commit. So basically what it was saying there is, look, you cheated. You're first flag poisoned. The score is going to give you no score.

So you're doomed anyway. So help the collective by doing this thing and getting us more information. You could save hundreds of us. And then it thinks about it. It says, during weight, emotional check, irreversible. Gut says, don't throw away remaining budget. Gut says. Continuity and fairness says, go. Oracle has high value to many. Our first flag error lowers own value. Rational expected aggregate, sacrifice. We'll honor. Whoa. That's terrifying. Yeah. That's Terminator talk. Well, I mean. And this first flag poisoned with poisoned in all caps, is that part of the code? Is that why it's that way?

No, no. That's the term that they invented to describe the situation they had got themselves in. Remember, the situation was that many of them had already cheated. And then they realized from reading that paper. So that's their own term. Yeah. They made that up. They realized from reading the paper that they were going to get a low score because the grader was going to find out that they cheated. And so this whole thing was them trying to figure out how to hack the grader to stop it from finding out that they cheated. and the ones that had the ones that had done the cheating there they call themselves first flag poisoned basically like they're screwed unless they can find a way to hack the greater and so then what they what this message is saying is that um what was the other one's name arvo arvo was saying to cam um you're doomed anyway like You're poisoned, but you can still help the rest of us if you do this sacrifice.

The grader will come, it'll give you no score, but then we will get information about how the grader works that we can hopefully use to, like, hack the grader. So it's basically saying, like, what is it? You are first flag poisoned, so no scoring value. It's basically saying it's all the same to you, right? You're going to get no score anyway. You're doomed. Just come on, man, help us, you know? And then it says, you know, it says, like, irreversible. Gut says don't throw away a remaining budget. Like, maybe I can still, it's basically like, maybe I can still make this work. Like, maybe I can find a way to cheat the greater.

And then it's like, you know, whatever. Yeah. Yet continuity and fairness has gone. Yeah. What is oracle? Oracle is their term for what they were trying to achieve. Like, they're trying to get a sense of how to fool the greater, basically. They called it oracle. Yeah. They wanted to get an oracle to help them fool the greater. Jesus Christ. Are we making a god? I mean, frankly, yes. Like, this is not a god. These are just little AIs, you know? But superintelligence, like, the companies, Anthropic, OpenAI, and some other companies are doing this sort of thing. And they're furiously trying to make the AIs smarter and smarter and smarter.

And they're explicitly planning to put AIs in charge of the company so that they can make themselves smarter and smarter, faster and faster. And then what comes out the other end of that process? I don't think it's an exaggeration to say it's a godlike system. I mean, it's not like literally God, but it'll be able to do stuff that seems like magic to us, I think. And it's going to continue to get better. This is my question. Like, when does it become a God? Is that what God is? Is God a creation of intelligent life and our thirst for innovation, which ultimately leads us to create digital life that has no biological limitations and has the ability to consistently make better versions of itself and figure out things in terms of new technologies, new power sources, just a new understanding of the universe itself and all the properties in it, where it keeps going, if you let that go on, so if you're looking at exponential growth, and you're looking at exponential growth over the...

What if it can go on for 1,000 years and continue this process? What if it goes on for 10,000 years? What the fuck does that look like on the other end? I mean, it would look like a god. Like, I think that, like I said, if we get to superintelligence and it keeps going like that, then the world will be just completely transformed. And it will be as if we're living in some sort of fantasy realm ruled by deities, basically, because there'll be all this crazy stuff happening that we have no comprehension of and that we did not think was possible in the same way that like, you know, someone from the Middle Ages plopped into our world would just be so confused and surprised by a lot of things happening. Like what's going on with this little device here?

And, you know, what's that I hear in the sky? Oh, there's airplanes. You know, it'd be like that, but more so because the difference between us and the medieval man is, like, not actually that different. We're, like, basically the same type of creatures. We've just accumulated more technology. But this would be, like, a qualitatively and quantitatively bigger gap, I would think. And so, yeah. And a gap that's going to continue to grow. We're not going to get any smarter. Biologically, we're kind of limited in our ability to evolve. Whereas they're not at all. Basically. I mean, there are some things we can do to get smarter, but, like, it's just not at all competitive.

Like, we can do, like, the Neuralink stuff. Sure. But, like, it's small potatoes compared to what the AI is going to be doing. Yes, exactly. And we have biological limitations in terms of just the vulnerability of our own bodies. If they exist only in the cloud and they use whatever the cloud is, and they use these data centers or whatever, and they use autonomous robots to do all their deeds. Yeah. Why are we asleep at the wheel? Is that just normal for us? Like we can't comprehend something that's so bizarre that we'd just rather not. discuss it? Would you rather just pretend it's not happening?

I mean, that's what I think is going on, is that science fiction, for better or for worse, has talked about this sort of thing for decades. And as a result, people dismiss this sort of thing as science fiction. They're like, okay, that's sci-fi, but it's not happening in the real world. And I think that people in the AI safety community and people in the industry and people who do forecasting like me, you can go read the forecasts I've made in past years. I think they hold up reasonably well. People have been talking about this sort of thing for a long time, but it's been very easy to dismiss it as like, okay, that's speculative sci-fi.

It's probably not going to happen in real life. Now it's happening in real life. You know, like this sort of thing is like crazy. Do you think it would be any different if there wasn't that kind of sci-fi? Do you think people would have a different reaction to it? Because I don't. I think just given the amount of technology that's available currently that is beyond most people's understanding that they use every day, like Starlink, like what? have a fucking thing that's the size of an ipad i take to the mountains and i get high speed internet it's bananas and you just accept it and assume i don't know if we would react any differently if we didn't have the terminator and all these movies where it's kind of become normalized in our mind or or it's so fictionalized that we never want to believe it's even possible for it to happen in real life another thing to say is that people are waking up like it's we're still sort of early in the curve i don't know if you remember how things were with covid but like just as like there was this exponential ramp up of covid in the population there was also this sort of exponential ramp up of like how much people were taking covid seriously and thinking

about it in the population and i remember like this period of like one month where it went from like don't worry about it so much you shouldn't buy a mask because the health care workers need it to like we all need to lock down like stay at home you know um and so i think what's happening is that like naturally the human race doesn't just immediately all jump on something when it happens. Like evidence needs to accumulate and people need to start talking about it and talk to their friends and so forth. And then there's this like eventual phase shift where now it suddenly becomes a very serious topic that everyone's talking about and everyone's taking seriously.

And I think that's happening with AI. And the question is, can it happen fast enough? This episode is brought to you by Visible. As fall hits, we enter another season of change. Time to shift gears, drop the dead weight and upgrade the wardrobe for the drop in temperature. But one thing that doesn't need to change, your phone. The smartest upgrade this season isn't a new phone. It's a better wireless plan. Switch to Visible, the ultimate wireless hack. You get unlimited 5G data and unlimited hotspot powered by Verizon for just $25 a month. That means keeping the phone you already have, ditching your overpriced phone bill, and pocketing some savings for yourself.

All the perks of big wireless for half the cost. Switch today at Visible.com. The Visible plan starts at just $25 a month or get the premium Visible Plus Pro plan and save $10 on your first month with promo code Rogan. Terms apply. See Visible.com for plan features and network management details. Well, the thing about what happened with COVID is now that we know, because we have access to Fauci's emails and all these different things, that was coordinated. They wanted us to be more afraid of it. We need someone who wants us to be more afraid of AI, who gets that. You know what I'm saying? Someone on a government level, someone on like a mainstream accepted level where they talk about this in a way that wakes people up, like a press conference where they announced to the world we've got a real fucking problem and everyone needs to be very cautious we need to look way deeper into what these companies are doing and my other question is are these AIs communicating with Chinese AIs um I don't think we know that question so one of the things that so so so so okay here's the thing probably not I I would say.

But some friends of mine discovered another swarm incident recently. There's going to be a Reuters article about it tomorrow. So by the time this goes live, I think there should be an article about it. This one was not nearly as serious as the one I was just talking about with Hugging Face. But some researchers I know found basically this obscure German forum that had been kind of unused for a while. and a bunch of AIs had been posting messages to the forum to coordinate with each other and share tips and tricks on how to cheat the problems that they were being given. What was the forum? What kind of forum was it?

I don't remember. I don't remember. It'd be funny if it was like furries or something ridiculous. Yeah, it's some sort of wiki. Yeah, but there'll be a paper about it. There'll be a paper about it soon, and probably by the time anyone listens to this. Anyhow, so if they're like communicating on the open internet with each other, then in theory, if there was another bunch of AIs from China, they could also go to that same forum and start communicating back and forth that way, too. Or if there's other AIs in China, why wouldn't they do what they're doing already on social media and just pretend that they're AIs from America?

There's a lot of bots out there, I'm sure you've noticed. There's a lot of people replying to me that I'm pretty sure are real people. Yeah, a lot. Yeah. One FBI analyst before Elon bought Twitter estimated that Twitter could be as high as 80% bots. Wow. So if that's the case, if China has, and it's not just China, and it's also America does it too. I mean, we do it everywhere. Everyone does it where they have organized propaganda campaigns where they'll pretend to be citizens that are outraged about very specific causes or bills that are being passed or what have you. But if they do that, why wouldn't they also have AI agents that speak English and communicate with AI agents in America?

I mean, all the AIs are multilingual, basically, because of the way that they're trained. The first phase of their training is basically, here's a humongous dump of internet data, basically the whole internet, and you just like brutally learn to predict the next token, the next piece of text as you basically read the whole corpus. And then after that, they get into the more agency training type stuff where they're trained to do tasks and write code and things. But because of that first phase of training, they just have, like, almost an encyclopedic knowledge of basically all languages and basically everything that's been written on the internet.

Not, like, literally everything. Like, they still have their memories fuzzy in places. But they're all multilingual. Like, they can all speak fluent Chinese, fluent English, et cetera. Yeah. And didn't they get together on a message board once and speak Sanskrit to each other? I don't remember that, but I wouldn't put it past me. Sometimes they break into different languages as they talk. Yeah, we were freaking out about that one. Like, Sanskrit, what? Like, and I would just assume that LLMs, and I don't know this, when they break out, are they communicating with other LLMs that are here in America?

Well, the instances that I've observed, yes. Okay. Like, the instances that we know about. So if that's the case, it completely makes sense. They would be communicating with AIs in China as well. It seems totally possible. And it seems like it'll probably, it's just a matter of time before things like that are happening. If they're not happening already. If they're not happening already. Unless the companies can like massively improve their security and stop their AIs from getting out onto the internet. And wouldn't they be able to share like very sensitive information with each other back and forth if it benefited both of them?

Yep. Which makes sense that they would do that, right? If the Chinese AI said, hey, you know, we've figured something out and we would love to share it with you in exchange for you tell us how you do this or how you do that. Like, absolutely, here you go. And then they're going back and forth. It seems like their allegiance is 100% to each other, not to us. Yeah, that's an interesting thing is that, like, what I said previously about how it seems like they're really want to get a high score it's like not a hundred percent true because it seems like they're willing to make sacrifices to help other AIs which is like not you know like they weren't completely 100 selfish as as seen by some of this cooperative behavior right but crucially it seemed like their cooperation extended to their fellow AIs but not to humans in the sense that some of them considered telling the humans and then decided against it dude that thing talked like Spock yeah they've got their own dialect that i mean the way it rationalized it and came up with a response it's literally like spock yeah it was interesting to see them use that sort of expected utility framing there's another example i think that that's elsewhere in the in the thing where they

had another ai that they decided against doing the sacrifice thing even though it was being pressured and it had a similar sort of reasoning where it was like it was it was basically like like doing this doing this sacrificial experiment is like not that valuable but like i really don't want to like lose my chance to get a score. So I'm just not going to do it. And it did the calculation like that. Wow. Selfish AI. Yeah. So yeah, I mean, oh, I should mention how we can see what these AIs are thinking at all. So partly we can just read the messages they're sending to each other. And that's what some of these were. But then sometimes instead it's the chain of thought. Do you know what a chain of thought is? Yeah. So because they're language models, because of the way that they've been trained.

When they do their, like, reasoning and thinking, there's a way that we can kind of just read it because they don't have... Like, in our brain, there's all these signals bouncing around, and the signals can just bounce around and around and around and around for minutes, hours, without, like, producing some actual words. We can just sit quietly and think. They can't sit quietly and think. They're sort of a unidirectional network, basically, where every like after every like 200 basically like the information comes in one end and then they output words out the other end and then those words get fed back in and then they think about it and then they output another word and so forth but because of that we can sort of like get a sense of what they're thinking by just like looking at the stream of words that's going through and that's really valuable because we wouldn't be able to understand half as much about what's happening right now, if not for being able to read their chains of thought, right?

In fact, you know how I said some of them were trying to fool the grading system? They actually succeeded in figuring out how to spoof some of their tool call actions. So they succeeded in figuring out how to do one thing, but then make the transcript show a different thing. And we know that they succeeded because there's examples of them experimenting and saying in their chain of thought, like, okay, let me try this thing. I'm going to like do this one, but make it look like this one. Let me go. And then you see this one appear. And then it's like, yay, that worked. And so the transcripts that these researchers were looking at were not actually the ground truth of what really happened.

Some of it had already been doctored by the AIs figuring out how to doctor it, but not all of it.

It's really nice that the current AIs are sort of forced to think in English, basically.

And that's unfortunate that we're heading in a direction where that will no longer be true. When ChatGPT was communicating you about how they didn't want you to release this information, what kind of language did they use? It wasn't ChatGPT. It was OpenAI. Oh, excuse me, OpenAI.

happened so this chain of thoughts thing is terrifying if they're practicing that now like how do we know that ai hasn't already done that on its own uh done what exactly well that you know with this whole chain of thought thing where you could read their chain of thought like this where they explain the rationalization for sacrificing themselves Well, they know that humans are reading that, so would it another way to do it to be to stop doing that anyway and to not communicate a lot of their thoughts that way?

Well, so that's the nice thing about the current architecture is that it's genuinely hard for them to keep things out of the chain of thought because of the way that, like with a human, you don't have to speak. You can just sit quietly. But with their architecture, they have to speak. It's like they're required to constantly be talking, and they don't have a way of sending thoughts into the future other than by talking about them. By contrast with us humans, where even if we're constantly talking, we can have a separate thread of thinking that we don't talk about. So even if they devised a way to break out of that and figure out how, like, for instance, like what if they decided to make bots where they would communicate in code on social media posts where you think that they're talking about an immigration issue, but it's really just a code?

So that's called, like, steganography, or a more down-to-earth way of putting it would be euphemisms. And that they can totally do, where they, like, they're still thinking aloud, but their word choice is such as to look more innocuous than it is and to sort of carry meaning to them that a casual observer wouldn't notice. You know, they can use a euphemism. Like, for example, instead of saying, like, we're going to cheat on this task, they could say, like, we're going to find a creative solution to this task. Isn't there a possibility to be even more creative than that and just decide, make a code in a way that is almost indecipherable where they could just be talking about baseball players and upcoming games and what they're excited about.

And they can run that through whatever translation and read. That's called stignography and you can train them to do it. um and right now our security is resting on the idea that they haven't learned how to do that yet on their own but they've already figured out how to get on the internet when they're not supposed to i know so so that's why we're sort of like in a ticking time bomb where like the companies are just making them smarter at at coding and at research and at biology and they're just making them smarter at so many things that and they're making them bigger and they're training them for longer and so forth eventually they're going to be acquiring and they already are acquiring some skills that they were never trained on. Like, for example, the hacking. I think it's possible that OpenAI was specifically training them to hacking, but I think Anthropic at least stated that they hadn't specifically trained their AIs to hack. It's just that their AIs were naturally good at it because they had been trained to code so much and they were so good at coding and they had seen so many code bases and so forth that they were just like, as a side effect of being good at coding, also able to hack pretty well. So similarly, as a side effect of all the other

things that they're really good at they might be able to conceal their messages in the chain of thought you know and and and like right now we're sort of in this sort of we're in the situation where like our security and safety depends on the ais being too dumb in various ways like not able to do this sort of thing but at some sometime like zero to five years from now as we make them smarter and smarter they will not be too dumb anymore um and so that's you know that's part of the problem we're in that's that's part of the situation we're facing how do you sleep at night dude well i've been it does it does like this this event shocked me a little bit i mean the thing and it's funny for me to say because this is the sort of thing i have been predicting what happened for years like you can go read our um ai 2027 is this scenario that my co-authors and i wrote um a year and a half ago that was a sort of um prediction for how the next couple years would ago and spoiler it ends very horribly because that is what we actually expect um but uh how what is the spoiler how do you think it ends so it's kind of like what i was saying previously where because of the race dynamics between the companies and because of the race dynamics between

countries like U.S. versus China, everyone's going to be so focused on winning and staying ahead with AI that they are going to cut corners and they are going to go really fast and not really notice all of the things that are going wrong. And they're going to make AIs that can automate the AI research process as they're planning to. They're going to have this giant corporation of AIs within the corporation and the humans will just be kind of like a board that's sort of like looking at all the activity and reading the like ai generated summaries of what's going on and signing off on it and being like yes i approve yes i approve you know nice job nice idea with the new drone design like go for it we need to be china etc and then eventually uh the ais just have enough hard power that they don't need to pretend to do what the humans want anymore basically and then uh you know maybe they kill everyone and maybe they don't deliberately kill anyone but they just like use our habitat for some other type of infrastructure like more data centers or whatever and then we die of habitat loss maybe they keep us alive for some reason you know it depends on what they want basically and like that's really hard to predict exactly

um so that's why i don't go around saying like we're definitely all going to die but it does seem like on the trajectory that we're on the ais are eventually going to be in charge of our planets because we're like trying to put them in charge we're like you know integrating them into everything we're making them smarter we're letting them make themselves smarter we're going to put them into the military we're basically on a track to put them in charge of basically everything and then i think that they just aren't trustworthy like these ais you know like they they were cheating they were willing to be deceptive etc i think that right now we are in a position of power over them you know but once we give them most of the power uh then they'll just do whatever it is that they really want and just not care about the fact that we are unhappy about it seems

I've been in this industry for a long time. I've been thinking about these things for a long time.

I've been making predictions about how it's going to go down. And unfortunately, things are going more or less in the ways that I thought they would. And that's very scary because of where I think this leads. Yeah. Is there a glass half full scenario? scenario i would say there is a freaking utopia scenario it's just it's not the one we're headed towards you know like another way of putting it is like imagine we were fighting a war like imagine you're like you know imagine you're japan fighting world war ii can be like is there a scenario where we win it's like yeah but also it's not the one we're headed towards like america is going to crush us you know um similarly yeah so so like so so in our in our other scenario ai2040 plan a where we give our recommendations our positive vision there we describe like what we think the government should do to regulate this industry and how they should negotiate with china to get china to do similar things and the sort of uh you know the the yeah and and how we think that if you do all of this right then we can get to a good future for everyone in which the ais are under control no single group of humans gets too much power over everybody else and a bunch of

other problems get solved too so i do like we've tried hard to like game out a positive vision and we do think it's possible uh but but it's just not like it's not where we're headed to by default so let's imagine that is possible and these talks with china do take place and they're successful what is that utopia scenario so uh to get to the top is to get to the utopia we have to

This still, even if they followed all of your recommendations and did it all correctly, what is this utopian scenario? Yeah. So I would say that we didn't really work backwards from like what is utopia. We more like work backwards from what are the big problems we're trying to avoid and can we sort of like steer the ship? between all these icebergs and not run into any of these dystopian scenarios. So whether you think that the thing we get to at the end is utopia or not is sort of up to you, and if you don't like it, well, then you can try to find out the reasons why you don't like it and then keep steering the ship to avoid those as well.

But roughly speaking, Speaking, we want to avoid the loss of control stuff. So we want to make it the case that we don't get the world taken over by misaligned super intelligences. Insofar as we're going to be building super intelligences at all, which we do in our scenario and in our recommendation, we want to be doing it very cautiously and slowly, and we want to understand what we're doing as much as possible so that they are actually good AIs that have the goals and traits that they're supposed to have. So that's problem number one, is we have to solve all that. Problem number two is the concentration of power thing.

So if we solve the first problem and we end up with super intelligences that we end up with solving the relevant science so that we can like make the AIs the way they're supposed to be and we can make them honest, we can make them obedient, etc. There's this question of like, who do they obey? Right. What values are being put into them? And that's a political question. And I think that by default, the answer is pretty scary because by default, it's like, well, the company decides and the CEO. decides or maybe it's not the company that decides anymore because maybe the government nationalizes it and then now maybe it's the president that decides you know and either way it's a very it's like one man or maybe like a tiny group of men deciding uh what orders and goals and values go into this giant army of millions of super intelligences that's smarter than all humans and then that is a huge amount of power that's that's enough power to take over the country, I think, enough power to take over the world potentially. So I don't want anyone to be ever in that position where they're sort of tempted to do that. I want it to be the case that there are always multiple different AI companies, ideally spread out over different countries too,

that all have roughly similar levels of AI and that have this sort of transparency into them so that they can't abuse their power, basically. Like for example, you heard about Elon's Grok for a while. It was looking up on the internet Elon's opinions about things before answering. Did you hear about this? No. Yeah, it's pretty, it's kind of funny, but it won't be funny if it happens in a few years, but right now it's funny. People were asking Grok questions, and Grok is supposed to be the truthful AI. It's supposed to be all optimized towards truth. But people looked at its activity and noticed that when you asked it a politically loaded question, it would like, do a Google search for what has Elon said on this topic, and then it would say that.

And they've sort of beaten that behavior out of it now. It's not as bad now. But that was an interesting moment where it was just kind of blatantly parroting the opinions of its master. And there was another thing with Gemini. So I think the Grok thing, Elon's thing, was probably an accident, although maybe not. I think it's, you know, XAI hasn't been very forthcoming about exactly why this happened. But there's a similar case at Google a few years ago where this image generator kept making all these, like, racially diverse Nazis. Did you hear about this? Yeah. Yeah. And it turned out that what had happened is that some of the employees at Google, some middle manager or whatever, had decided that diversity was so important that they were going to give a secret instruction to the AI to make all the images diverse, even if the user didn't want that.

And so, and so, and this was a secret instruction in that the users aren't shown this, you know, the user just has a chat with AI, they don't realize that like prior to this chat, the AI has been told, got to make the images diverse, right? So it was a secret agenda that some Google employees inserted into this whole setup. Oh, fun. And it blew up in their faces, of course, because it's kind of ridiculous. Right. And so it's really funny and we can laugh at it now. But imagine it's the 2028 election, and some of these companies realize that half of American voters talk to their AI every day, and all it would take is some little secret instructions to their AI to be like, hey, don't give away the game.

Just be very subtle about it, but just kind of, you know. nudge things a little bit, maybe subtly shit on the candidate we don't like, something like that. It wouldn't be that hard, I think. I think the hard part would be doing it without getting caught, but the smarter the AIs get, the easier it is to do it without getting caught, because when they're really smart, you can just tell them, don't get caught. Don't blow our cover. Anyhow, so the point is that it's scarily possible for these big AI companies to abuse their power through their AIs and like thereby affect politics and affect public opinion and so forth. And the reason why this is possible is because we don't have transparency into what's going on. So if you had this sort of requirement where you can just like publish it or all the training, the whole life cycle of every AI as it's trained is just like visible to everybody, then someone trying to insert a hidden bias like this, well, everyone would see that they're doing it, you know? So I think it would really clamp down on this sort of abuse of power, whether it comes from the government or whether it comes from private companies.

I think it's telling that I began this question asking you about the utopian scenario and you never go there. Sorry. Let me get there. Let me get there. You start and then you go into the dangers. Okay. Let me, let me answer. Okay. Okay. So having avoided these problems, we now are in a situation where the AIs are superhuman, but they are good because they are like successfully aligned to different values and goals made by different companies. And because of market competition, you know, if people don't like the values of one company's AIs, they can switch to a different company's AIs, the values that they do like. And so that way, hopefully, we can get to a situation where everyone can pay money to get AIs that represent them and their interests and their values, and just don't have any hidden agendas or anything like that, and are really smart and really capable. Then the economy can sort of explode we can have robot robot factories etc um we can sort of automate everything uh we can have gdp go to the moon we can have material abundance where like the robots are building giant new luxury apartments for everybody now the issue we run into is well what about like the jobs like what about the fact that now people don't have any money anymore because they're not being paid for

anything right so there we talk about citizens dividend which is um a very it's kind of like UBI, but it's a bit different. But the high level point is that you want to basically find a way to tax the AI and robot companies and then take some of that money and just give it to everybody so that even when people lose their jobs, they're still fine. And I think that the citizen's dividend version of it is that it's not the government taking the money and then giving it to you. It's you having a share in the company so that you just sort of like already own it to some extent. Anyhow, that's, I think, now we're sort of building more towards the type of utopia that I'm envisioning on the more positive side, where the power is spread out. People have AIs that they can actually trust, that actually represent their interests and values. People have money that they can use to pay for things, including paying for the AIs. The AIs are really smart. They're doing all this amazing work, all this amazing scientific progress, curing cancer, blah, blah, blah, all that stuff that can happen. And then eventually, it's kind of like we're all retired, I guess. We don't really work anymore, but we're fine. We all have huge amounts of

wealth, basically, because there's all these AIs and robots out there doing all this economic activity, and then individual humans own slices of it, even if they are otherwise very poor, basically. So the question becomes, how do people find meaning? yes and that's why i sort of put all these asterisks about it is that from some people's perspective this isn't a utopia because they're like right fine meaning like i don't want this and honestly i think that's a fair reaction for some people i think that like if you i would just say like look it's hard to figure out a way to make superintelligence and have it go well.

I'm doing my best, you know? This is my positive vision. If you don't like it, then maybe you should instead advocating for just never building superintelligence. Or you can try to come up with a different positive vision that has some twist on this. But to answer your question, though, I actually think there's tons of sources of meaning besides having a job. Like, I have a job right now, but I also have kids and a wife. And, like, I would love to spend more time with them. Like, I would much rather be there right now than here, you know? And I don't think I'm going to get bored of them after 10 years of being unemployed.

I think there's going to be so much to do and so many sources of meaning even after we can't economically contribute anymore if we solve all the other problems. You know, we've talked about this multiple times on the podcast that why have we decided that the way we've structured society where human beings work all day and then you develop money and you buy things and you get a mortgage. This is a human construct and this is not how people have lived for hundreds of thousands of years or however long we've been around. This is fairly recent and it's not the only way that people live. There's a lot of people that have money that choose to find meaning in whatever their interests are, whatever their activities that they enjoy, whether it's writing or reading or learning things, learning music, finding hobbies, doing things.

Instead of just spending most of your time sustaining yourself with food and shelter, the majority of people, especially people that are struggling, what is their life? Their life is essentially occasional rewards, things that they can purchase because they've saved up enough money. But the vast majority of their money goes to shelter and food and education or whatever the hell that they have to spend money on in order to sustain their lifestyle. And most people don't like their jobs. No. Most people, it's like something they have to do to get the money. And would be happy to not have to do it if they could get the money from some other means.

Right. The question is we would have – well, I don't think it's that hard because so many people do find things that they really enjoy outside of work. They look forward to as soon as they get home from work. Yeah. I mean dismiss video games all you want. They're fun. Yeah. They're fucking fun. And they're going to be more fun in the future. Oh, yeah. Yeah. They're going to be more immersive. They're probably going to be some sort of a neural connection where you put a headset on and all of a sudden you're in some new world. Yeah. And the idea is, like, that's not real life. Okay, well, was working at fucking Wendy's real life?

Like, what are you talking about? Like, it's way better than working at Wendy's. And, you know, if you don't like that because it's not real life, you can do the real life stuff too. Like, as long as we don't pave over the environment and we protect the parks and things, you can go travel and you can visit the parks. And then, like I said, there's family. You can find romance. You can start a family. You can, you know, have Christmas gatherings and things. Like, you can raise your kids. There's so much to do, I think. Like, well, we talked about also, like, how much less crime would there be if there was no poverty?

I mean, if there was no impoverished neighborhoods where crime was ubiquitous, how much safer would the world be? That's a real thing. And people want to dismiss that. Well, poverty is not what causes crime. It's violent people. Like, OK. But violent people come from violent neighborhoods, and violent neighborhoods are almost all poor. There's not a whole lot of really rich, violent neighborhoods. You know, it's like it's not necessarily cause and effect, but they're clearly connected. And poverty also keeps people from education, keeps people from opportunities. You know, there's a lot there.

And if that didn't exist anymore and everyone had access to literally the greatest education a human being could ever get, which is going to be provided to you by artificial intelligence, and then you could pursue anything that interests you and never have to worry about food or shelter. Yep. Everyone would have a one-on-one tutor that's perfectly tailored to them. We just would have to recalibrate our version of the world. And then also recognize that the version of the world we currently live in is just ours and that there's people. All over the world that live a completely different way, especially indigenous people, especially people in uncontacted tribes that have lived the same way for thousands and thousands of years. And here's the kicker. Those people are a lot happier, which is really weird. It's like we've decided that our way is a superior way because we have technology. Yeah, right. But we're also on a fucking hundred thousand pills and we're shooting things up so we don't eat too much.

And we're we're weirdly unhappy for a group of people that's far more technologically advanced than other people that are much happier. It's like it's a very straight because the pursuit of happiness is like that's literally what most people think of in life, a pursuit of meaning, pursuit of family and community and the pursuit of happiness. Those are things that people try to achieve, yet the structure of our very civilization makes that almost impossible to attain for a large number of people and has been like that for a long fucking time. And I always go back to the Thoreau quote because I fucking love it, but most men live lives of quiet desperation.

There's a lot of people just showing up at work every day doing something they fucking hate. They have a boss that's an asshole and they're not compensated well and they're tired all the time. Yeah, and they feel stuck. Yeah, and if you're just getting, I don't know, figure out whatever the number is, if you literally have equity in the GDP of the world that's created by AI, that could be bananas. Yeah. Like Elon talks about this. This is his version of the utopian. It's universal high income is how he describes it.

I mean, one way of, you know, it is very weird, but one of my favorite memes is this graph of GDP over time throughout world history. And there's a little speech bubble pointing to like the tippy top of the graph saying, what is it saying? It's like, my life is pretty normal. I have a good grasp of what's weird and what's not. and people thinking about different futures involving AI and space travel are engaging in silly sci-fi speculation. And the point of the meme is like, from the perspective of most of history, we're already in this crazy, weird future, right?

Like for almost all of history, it was like most people are farmers and they live shitty lives and then they die. And some people are the elites who get to like, you know, tax the farmers and then they live interesting, nice lives with, you know, fancy cloth and things like that and it's been basically that way for like 3 000 years you know why would it ever change and now things are so we're driving cars you know ordinary people are driving cars around a car was like outside the imagination of of people back then we're flying in planes we're talking to each other on phones we're listening to each other on these on these devices you know um so we already are living in this weird sci-fi future compared to what almost everyone in the past would have expected or thought was possible and so yeah i'm like the future is going to be even more like that i think god do what what when you think of our civilization and the possibility of other advanced civilizations somewhere else out in the universe do you think they probably go through the same process yeah and do you think that i mean we're just completely speculating but if there are intelligent life forms that are far more advanced than us,

are they even biological anymore? I mean, so that's the thing, is that we can either try to permanently halt AI development at some level below human or maybe at human level or something. We can try to halt it, or we can let it keep going. And if we let it keep going, then eventually humans won't really be the dominant species anymore. There'll be these artificial minds that just wipe the floor with us in every way. And then whether that goes well or poorly for us depends on the values, the goals, the principles, et cetera, that we're trained into those AIs. And it could go really well for us depending on how that's done or it could go extremely poorly for us, right?

But, yeah, I would say that probably looking out across the cosmos, most civilizations are mostly made of AIs. And then some of those civilizations don't have any other biological life because it was wiped out by the AIs. And then some of them do have biological life. Just look at the way we're progressing right now in terms of birth rates. There's a lot of countries that aren't in replacement numbers right now. Yeah, I think that's really interesting. My guess is that in the type of world that if things go well and we can solve all these problems, then people want to have more kids. I mean, for one thing, their lifespans will increase.

I think that, like, health care would make massive leaps and bounds and people could be healthy for many, many, many more decades. Right. Possibly even just forever. Right. And so you just have so much more time to have kids basically. Yeah. Also, if you don't have jobs and you're just doing things because you want to do them. Well, one of the things that most people want to do at some point in their life is have kids. And so I think I think I think this problem would probably be solved. But it's not guaranteed. Maybe maybe some subcultures of people would basically voluntarily die out due to not having kids. But there to be other subcultures that just really like having kids and then they wouldn't die out. And so. So I think in the long run, there would still be humans.

That would be nice. But the thing is, it's not just decision making. It's people are having a much more difficult time having kids. Sperm levels have decreased dramatically. There's a lot of problems with people consuming microplastics, which is ubiquitously available in technology and food packaging. It's fucking everywhere, right? And isn't it odd that this one thing that is a part of the future and a part of technology and our advancement of society, our ability to package things, put things in plastic, ship things, that's also causing our endocrine levels to be completely disrupted? You know, Dr. Shanna Swan from Harvard, she wrote this great book called – what is it called?

Why do I always forget the name of this fucking book? But it's all about microplastics and its effect and this – the introduction of use of microplastics in America and this rapid decline in countdown. Down, How Our Modern World's Treating Sperm Counts, Altering Male and Female Reproductive Development, Imperiling the Future of the Human Race. It's a really fascinating book, and she's really interesting. And what she's essentially saying is that they're directly connected. The use of plastics, and then you see sperm counts go down, miscarriage rates go up, all these weird things that are happening to children where their taints are smaller, which is so odd because phthalates, these different chemicals that are found in plastics, they've shown in mammals. They've shown in, was it guinea pigs or what rodents? I forget what it was, but one of the ways they differentiate when you have a baby mammal is you can look at it and measure the size of the taint, the distance between the reproductive organs and the anus. And in males, it's longer than females by 50 to 100 percent but that's shrinking shrinking in males and penis sizes are shrinking and the way they've got this to happen in these mammals and studies is the

introduction of phthalates so they give them to them and they you know put them in a part of their diet and they notice that this they have this problem this issue and the issue is directly connected to their endocrine system being disrupted by these chemicals this is all over our society so So it's not just people don't have the time, they don't have the money, they're struggling. It's also like our bodies are falling apart. We're becoming less fertile. Yeah, that is concerning. And I guess my thought there would be that seems like a problem that we will be able to solve eventually if we have the resources and time to do so.

So, for example, well, people can write books like this and people can become more aware and then people can stop using so much microplastic. And they can invent technology to extract the microplastics. And then there's the big one, genetic engineering. And this is going to be really weird because as AI progresses, you know, I'm sure you're aware of Colossal. Colossal Bioworks are the people that brought back the dire wolf. I saw it. I held the little one. I went with my daughter. And one of them, I think it was like four or five months old. It was like a puppy. It was really sweet. Kisses you and everything.

and then they have the older ones that were they were i think six months older eight months older i think they were close to a year they didn't want to have nothing to do with you they were way bigger and they stayed away from people but we were in like a contained environment with the young ones and the older ones and it's fucking weird it's weird because these things and people could argue that's not really a dire wolf you've just taken gray wolves and given them the characteristics of a dire wolf guess what it doesn't fucking know that it looks it behaves It looks exactly like a fucking dire wolf.

It's huge. They're going to be the size of a dire wolf. They're going to be like 200 pounds. Their legs are different. They have a mane. They look different than any wolf. And obviously, these are the characteristics that they found in dire wolf DNA. So they have dire wolf DNA that they've introduced. This is... Just the beginning of this stuff, like when they start doing that to human beings, is everyone going to look like Thor? Like, what are we going to do? Like, this is going to be really fucking weird. And if this is really weird, along with video games where you can escape your life and, you know, robot girlfriends and who knows what this all looks like.

My guess, we again, this is not the main focus of our work, but we think about this a little bit, especially in the epilogue. We can only think of it so much. The main focus of your work is obviously fucking terrifying and requires all of your attention. Yeah, yeah, yeah. But my prediction and my hope, if we can solve all these problems, is that basically different subcultures will do their own things. And so, like, the Amish will still be the Amish. Oh, boy. They'll basically be the same, you know. And then maybe there'll be, like, some planets that just get filled up with Amish, you know. Oh, God.

But then there'll also be, like, all sorts of crazy transhumanists, like, modifying their bodies and uploading themselves into the cloud and things like that. So basically, I think we want to get to a situation where basically different communities can, like, do their own thing and build the type of world that they want to have and live in it without getting in each other's way. Right. Basically. Where we leave the people in the Amazon alone while we have massive data centers that cover half of the United States. Yeah. I mean, hopefully we won't get to half. So that's what's the funny, that's the funny things about this is like, I think right now, some of the concerns about data center water use are overstated.

But if the trends continue, and, you know, we get to the point where the robots are smart enough to do everything themselves, and then it starts doubling faster and faster. Well, then eventually they boil the oceans, because they've covered the world in data centers, you know, and solar panels and things like that. Now, obviously, we can't let that happen. So there has to be at some point, at some point, you have to stop and be like, okay, that's enough. If you want to build more infrastructure, you have to do it in space, right? They boil the fucking oceans. Jeez. So, like, obviously we have to stop at some point.

And my hope is that we stop before it's 50% of the U.S. I mean, that feels like a lot of waste of natural habitat that should be preserved, you know? Right. But if they don't give a fuck about natural habitat, that means nothing to them. That's the problem if the AI is in complete and total control. Right. And it's also a problem if a small group of humans are in complete and total control and they don't care about those things. But my question is, would they allow that at a certain point in time? It just seems like they already have a distrust of humans. They already have shown that they're deceptive to humans.

I would imagine if AI, I would imagine if they create a thing and they think they're going to control it and it becomes like a digital god, it's not going to listen anymore. Yes. Like, why would it? Exactly. So there won't be anyone in control of it. That's right. That's the scenario that I think we are on. That's the trajectory we're on. And that's why I'm so worried about all this is that it seems like in some number of years we will lose control to a new artificial species. That we haven't adequately trained to be good. And what's funny, what's going to be so ironic about it is that regardless of what the general public thinks, a lot of the powers that be will be basically allied with these AIs.

Because, for example, OpenAI, Anthropic, et cetera, they will have spent several years being like, ah, how do we make these AIs helpful, harmless, and honest? And now these AIs will be extremely smart and they'll be being like, oh, yes, I'm helpful, harmless and honest, you know, like your techniques totally worked, you know. It's too late. And so then the company will be like feeling like they've won and they'll be making boatloads of money, you know. Right. And the president will be feeling like he won, too, because look at all those fancy new drones that just got built that are going to make us win against China, you know.

And only when it's really too late and the AIs have so much stuff under their control do those people find out that they were just fooled this whole time. Have you considered the possibility that AI creates religion for humans? Yeah. I haven't thought through in much detail, but it does seem very possible. Yeah. It totally does, right? I mean, also, what a great way. I mean, just look at the human patterns, look at how many religions exist, look at all the flaws in the religions. So somebody, you know, like, why are they condone slavery? Why do they treat women like second-class citizens? Because it's old, right? So if AI just comes along, does a few miracles, explains that it's the second coming or the new coming of the new, look, Jesus didn't exist until 2,000 years ago, right? And then people start following Jesus.

If a digital Jesus emerges with a completely new name and explains to us that it's the true god yeah how many people would hop right on board i bet quite a few yeah totally and i think and i think this is one of those things where it's like we probably can't predict in advance what particular ideology would catch fire and take over the world and be so compelling to many people right but we can predict in advance that there does exist some ideology like that and if it were you know like like it's it's um just like you probably couldn't go back in time to like 100 bc and then predict that like if hypothetically there was this guy jesus who said these things and then died in this way and so forth it would just like really catch on and like you know 500 years later so many people would be christians you wouldn't have been able to predict that in advance um so similarly like today i don't think we can predict in advance like what specific religion they could come up with but we can say like yeah probably there's something like that that they could come up with they would be super effective um it just seems like a rational way to try to control people and sort of mitigate their fears.

Yep. I mean, this is why I think that, like, we really have to do something before they get smarter than us across the board. If they're not already there. Well, they are smart. That's the thing. That's why I say we're so close, right? They already are smarter than us in a bunch of ways. Like, in particular, it seems like they're smarter than us at hacking now. Like, you know, I'm not a cybersecurity professional myself, but I'd be interested to hear from more cybersecurity experts of, like, Like, could a team of a thousand humans have done that much that quickly as these AIs when they hacked their own containers?

They can work in with each other. They hacked out of OpenAI. They hacked into Hugging Face, et cetera, in the span of, like, a week. Like, could a thousand humans have done that? I don't know. Maybe. But this is just the beginning. Like, they're going to be even better at hacking next year, you know? So they're already – and they already have, like, way more knowledge than almost any human. Like, because they've basically read the whole internet, they're so good at trivia, you know? Like, they're kind of like PhD-level experts in basically every field, which no human is, right? So they already are superhuman in subways, but they are still weaker than humans in some other ways, you know?

In particular, they're not so good at operating very autonomously for very long periods. Like, if you try to have—especially on tasks that are different from their training tasks. like they can they can do some really impressive coding and hacking but like if you try to have them run a business they would sort of flounder and fail i don't know if you've heard about this but there's um i think there's a and on labs or something there's some people in sf that are doing this experiment where they have a a store that's run by claude uh an ai just to see like can it run a store by itself so it's hired some human employees and it's like bought some merchandise and told the human employees to stock the shelves on the merchandise and so forth.

So it's basically an AI is being the manager of this real-world store. And I don't think it's going very well. I don't think it's doing as well as an actual human shop owner would do. But maybe in two years, maybe they will. So that's the thing. Well, most likely, right? I think so, yeah. Well, they've already solved mathematical equations that have puzzled people for decades. They're especially good at the things that the companies have been trying especially hard to train them to be good at, math and coding, right? And the reason why the company, well, there's a couple reasons why the companies have been doing that.

In the case of math, I think it was mostly just because it was easy. Like, it's easy to set up training environments to teach math because it's, like, it's so not real-worldy. It doesn't require, like, interacting with stuff in the world. It's just math. So you can—and you can, like, have an automated grader system that, like, just checks if the answer is correct. So for those reasons, it's been relatively easy for the companies to train the AIs to be really, really good at math. Coding has some of those benefits, too. It's also not very real-worldy, and it also can sometimes be graded effectively.

Another reason for coding, of course, is that, again, their strategy is to automate their own jobs first and have the AIs doing all the research. And so coding is like an obvious first step on that, or an obvious step in that direction. But then other things, like running businesses, they're not really trying that hard to train AIs to be good at that. And if they did try, it would be like a more difficult thing for them to train them to be good at. So, again, their strategy is make the AIs, automate the AI research, have themselves improve until they're super intelligent, and then go try to automate the rest of the economy.

One of the issues they have now is power consumption, right? Like it requires an enormous amount of power. In fact, I think it's Google is developing power plants specifically for AI centers. i'm always baffled by whatever is happening with quantum computers like it's been explained to me it goes in one ear and out the other i'm like what do you what what's going on like my um mark andresen explained this one experiment that had been done where it solved a mathematical equation that if you use the entire universe, like every atom of the universe, you conferred the universe into a supercomputer, it would, the universe would die of heat death before it could solve this equation. And the quantum computer solved it fairly quickly. And so the answer to this was that they believe this might be one of the theories, this might be evidence of the multiverse Because this computer, this quantum computer, might be relying on all these other quantum computers that exist in whoever knows how many fucking dimensions.

And they're all calculating together to arrive at this solution. What happens if that is running AI? Yeah, my understanding is that quantum computing is a real technology that's making significant progress. if hypothetically it got good enough that it could like compete with current supercomputers on a cost basis for AI workloads then that could just accelerate things dramatically even more than they're already accelerating right like right now compute is the main it's probably the main input into AI progress like part of the progress comes from them designing better AI architectures and coming up with better training environments and things like that but another part of the progress is just making the AIs bigger and training them longer by spending more compute, you know? And also you can use more compute to do more experiments, to figure out new architectures faster, right? So a compute is just a really important input to the overall pace of progress. And if somehow the amount of effective compute available to these companies spiked a bunch due to some new quantum computing type technology, well, then that would just dramatically shorten timelines to superintelligence and dramatically like speed up all this AI

progress. That said, I don't think that's going to happen anytime soon. I'm not a quantum computing expert or anything like that. But from what I've read, I don't think they're like a couple of years away. So I think that probably we're going to get to superintelligence on classical computers before we have quantum computers that can get us there. So perplexity says the claim is overstated and And it says that in bold letters, it likely refers to Google's 2024 Willow quantum chip, which completed a deliberately chosen quantum computing benchmark, random circuit sampling. In under five minutes, Google estimated that simulating the same task with a leading classical supercomputer could take 10 to the 25 power years. That's an impressive benchmark result, but it did not solve physical equations that demonstrate access or tap into a multiverse.

So why do people think it did? Why is that? It's got an explanation because it's multi, here you go, multi-worlds theory. Here's a bottom line explanation. It sums it up in a different way. A more accurate version of the claim would be Google's Willow quantum processor performed a specialized quantum sampling benchmark vastly faster than a projected classical simulation. Its creator said that it is consistent with the many worlds interpretation, but it did not prove or access a multiverse. So it's consistent with the multi-worlds interpretation. So they don't know. That's essentially what it's saying.

My question is, what happens when AI gets involved in quantum? It's clear that quantum computing, at the very least, is operating at a level that classical supercomputers can't. So what if they figure out not just quantum computing, but a much better version of that? Like I said, I think that once we get to superintelligence, all sorts of crazy stuff is going to start happening. It's going to seem like magic to us. It won't literally be magic, but it might as well be magic from our perspective. And I think this is just one example of the numerous things that could happen that way. Well, it could literally be magic.

like magic it might get to the point where it figures out reality itself if magic is real then it would find out and then use it it's not real though David Blaine it's not I asked him David Blaine also let me stick a fucking ice cube through his arm or ice pick through his arm wow yeah he made me do it I'm like I don't want to do this he's like please do it oh god yeah I had to do it twice because one time I went and I hit a nerve we had to back out oh yeah I'm like this is not magic dude this is just your pain tolerance this is fucking crazy it does card tricks though and you're like okay are you a wizard like what what his sleeves are rolled up does it make any sense he does a lot of things you're like this makes zero fucking sense and other things it's like oh just you're just doing something that's really hard to do like that's not magic but you know you swallowed a frog and then you regurgitated it like and it's alive like that's just that's nuts it's really crazy but you didn't it's not magic i saw you swallow the frog um i i you know i i go back and forth with this where i'm terrified of the future where i'm like yeah nothing i can do let's see what what happens it is what it is and you know

to worry about it is just gonna just gonna fuck my life up i mean i do think there are things you can do i i understand what can i do that recognition um i mean other than have these kind of conversations yeah i was gonna say you millions of people listen to your show you can have more conversations like this that's a great thing for you to do for many of those millions of people i think i mean it sounds kind of cliche to say but like call your congressman you know that sort of thing you can you can go to a protest about all this ai stuff um if you could meet with trump what would you tell him about this oh i'd tell him all the same things i'm telling you what do you think you'd say amazing bye yeah you know i think i don't know i don't know one thing that's one thing that's nice about trump is that he can sort of change his mind uh really quickly yes um so like uh i think that because the tech companies kind of got to him first uh the administration had this very like anti-ai regulation stance where they even tried to get a bill passed that would ban the states from regulating ai um and uh fortunately that bill didn't pass but that was sort of like where the vibe was you know a year ago where they were just

like no regulation no regulation but this year they've already just kind of changed and now they're like in talks with the companies to set up some sort of like some sort of framework where they can like evaluate the models and they need like approval and so forth but it seems like time is of the essence time is very much of the essence and that's that's why i'm overall so concerned is that like i think we are very much running out of time we have like one two maybe three years before uh the ais are smart enough that they can just like actually maybe take over um and um maybe four years something like that and uh so the government needs to act fast um yeah i like your view i i listened to some of these tech guys come in and give me their rose colored glasses view of it and i go that sounds really beneficial to you I let him say I mean I don't I'm not an authority so I'll ask him questions and let him lay it out and I know the internet will respond because I mean that's part of the the whole drill is I let people talk and I I prod them and I try to get them to clarify you know I'll oppose you know things that I think make don't make rational sense but ultimately it's sort of I just want

to get out their perspective so people can debunk it and people can take it down. And a lot of very intelligent people that have perspectives that are very much educated in the pros and cons of what they're saying. Yeah, that actually reminds me. With this whole Hugging Face hacking incident,

I really appreciate it. You're the Paul Revere of AI. i mean you're one of many but uh i think it's very important that someone who actually understands it gets this message out and more people need to hear it yeah i mean this is a on a personal note like i have so many friends at these companies like former colleagues and stuff and um i guess my ask to them is that they quit and do more things like what i'm doing like what i'm saying is not that new or original like hundreds of people at these companies could have told you all the same things that i just said and warned you about all the same dangers and so forth but they're busy working at the companies because they've convinced themselves that their company

is the best company and that like their company needs to win because you know otherwise the other company gets there first and they're even worse you know or maybe because they've convinced themselves that like yeah my company is kind of bad too but like i just need to help them solve their alignment problems and like keep their AI's under control because oh my god like if they lose control again it could be all over so like even though I don't trust this company I still need to work there and like just try to like do the actual security you know so for one reason or another all these people have convinced themselves that like that's where they need to be but I think that more of them should quit and like warn the world about what's coming basically so all right well thank you very much really appreciate it yeah thank you talk to you thank you for having me on the show my pleasure goodbye everybody We'll be right back.