Max Tegmark: We’re Building A Replacement Species

You go back to Ancestral Home and pick up the golden medallion. It used to be on your Linux signature or in your status. I remember from 20 years ago when we first, 30 years ago, what am I talking about? 30 years ago, we started collaborating together. Can you believe it? And we wrote a couple of papers together. And you used to sign off, you know, you're waiting for the call from Stockholm or something. But I know you're going to get it, Max. I know you're going to get it. But let's start with the Navier Stokes. I was actually working a little bit on this problem with a PhD student of mine who's graduated now.

So it was extra interesting for me when this story broke. Yeah. How did you react to it? Is it like a Higgs boson, bicep two moment as you wrote, you know, goodbye universe, hello multiverse on March 17, 2014 on your blog? Was that this kind of moment like that? I would say it's a little bit smaller, but it's still huge. First thing first, you know, they have not actually solved the problem that you get a million bucks for from the Clay Foundation.

Max TegmarkThey've taken and solved an easier problem, which is still super impressive.

So, you know, the Navier-Stokes equation is the physics equation that can describe the water flowing in your bathtub, making a nice little whirlpool, like my three-year-old son loves to look at, or a tornado, or all sorts of other fluid dynamics. And the question that's being asked is, is it possible to start with a quite normal situation and then start getting the fluid going infinitely fast, which seems completely nuts and would violate the known laws of physics? And they proved that they can do it, but with a kind of cheating where you're allowed to keep pushing on the stuff. so you know when when leo gets this vortex going in the bathtub with the water going out the drain he's not pushing the water it's just flowing there's some gravity trying to pull it down but there's nothing pushing it around to make it spin but they are doing that they're allowed to have a force that acts in some smooth way everywhere in the water so they can actually make it spin faster and faster and faster but not not infinitely fast and then the trick is this you know you know if you look at a figure skater right and they they're spinning and they pull their arms in they will spin faster that's because of angular

momentum being conserved of course and that actually implies that if you could take the figure skater and they could like shrink themselves together to infinitesimal size to basically zero size they would go infinitely fast the speed you'd have to go would be infinite But if you look carefully at that vortex in your bathtub, you see there's actually no water in the middle at all, because you have the centrifugal force pushing the water out and instead you get this neat funnel shape. Same with the... Tornado, if you look close, it's actually hollow in the middle, right? That's why it looks like a funnel. And so there's almost no air in the middle, almost a vacuum, which is where you can get sucked up into it.

Yes. And so that kind of solution is never going to win you the million bucks from the clay foundation, right? Because it's only when things actually start spinning and get collapsed together to zero radius that it happens. But then you have this ace up your sleeve called viscosity. If you take a real fluid, honey or molasses or something, it's a little bit sticky. And that lets it actually dump some of its angular momentum and make things spin in principle in even smaller circles. This is how black holes eat lunch, for example. You know very well that when black holes start snacking on a cloud of gas the gas will typically pull in and it'll be spinning faster and faster and so it flattens into kind of a pancake shape and if there were no viscosity if there were no stickiness in this gas you would have the same problem as in your bathtub that there's like nothing there in the middle for the black hole to eat but there is some viscosity the gas bumps into gas that's farther in and and so on and then the black hole actually gets to ultimately eat all of that stuff up so so this is the heart of the physics challenge or can you make some sort of spinning vortex eventually spin infinitely fast by making it contract by using them using some of this

stickiness but not too much stickiness either because you don't want to like get rid of all the motion then you're never going to go infinitely fast and what they've now proven is that yes you You can actually do that if you're allowed to keep pushing it along with this weird external force. But to get the million bucks, you have to prove that this can happen with no external force. But you can actually just take your fluid and get it started somehow, and then stand by and watch this crazy singularity emerge. Does that make sense? Yeah, and to do so in a finite time. I mean, the thing that's interesting to me is the Navier-Stokes equations don't involve any thermodynamic considerations.

Like, we know the laws of physics are, you know, reversible, at least, you know, CPT invariant. But, you know, if you would see a vortex, you should also see an anti-vortex. You know, in other words, all the... fluid kind of, you know, going the opposite direction and then creating some kind of anti-whirlpool, for example. But we never see that, just like we never see, you know, rings of water converging and then making a stone, you know, fly out of it, right? So why doesn't turbulence or, I mean, turbulence appears, but why doesn't thermodynamic dissipation appear in the Navier-Stokes, you know, equations or their solution?

So let me say two things to this. First of all, if you started with just water, like, perfectly at rest in Leo's bathtub, but it's not going to be able to start spinning either clockwise or counterclockwise for exactly the reasons you say. There's no angular momentum to start with, so there's never going to be any. But, you know, Leo has always been splashing around a little bit. There's some little random motion. So on average, there's going to be a little bit of spin. It's just like when our solar system was formed, you know, the big gas cloud that contracted and made our solar system wasn't spinning very much, but it was just randomly spinning a little bit on average.

And then when it contracted a whole bunch, it became very obvious that the solar system is spinning a lot, and the sun spins around every month, and Earth orbits the sun every year. As long as you don't start at rest, you start in some sort of messy, random situation, you're always going to be able to get a whirlpool going one way or the other. Second part to your question, thermodynamics. so just like all other laws of physics are just approximations we believe of something more profound same thing with the Navier Stokes equation and in that case we know one huge thing we're glossing over which is that all matter is made of atoms so it doesn't make sense for example for in real life for a a gas to suddenly be spinning down and the radius go to zero because then that will be smaller than the separation between the atoms as soon as you're the spacing between the atoms is bigger than what you're modeling the scales you're modeling you know things are broken down so even if if this uh million dollar conjecture is true you would never actually get anything infinite and instead but you would you might be getting things going crazy fast and then those crazy fast moving atoms would bump into other atoms and you would get heat dissipation you know thermodynamics

kicks in just like you said the thing that was also very interesting to me and made me want to talk to you uh you know first and foremost, was this issue of, you know, kind of the, a little bit of a scandal. I mean, I don't like to talk about people and their motivations, but it did seem like there was something a little bit nefarious that was going on, perhaps the allegations, if they're true. And I've offered to speak to, you know, Sebastian at OpenAI, and he follows me on social media, and likewise. So I'd love to have your perspective. And I think it comes concomitant with the hugging face incident in some sense.

I got these messages, I'm sure you did too, from Claude or Anthropic, from OpenAI. Actually, you probably have the good stuff. I just got there normal, you're a scientist, professor, sign up for OpenAI for scientists, and then somewhere in the 80-page terms of service, it says they could train on my data, unless I followed this very arcane thing like sneaking out of my kid's Lego pile without getting my foot stubbed. And I just didn't do that, And so I never actually use the OpenAI version. I do use the Claude version. But anyway, it seems like they may have, you know, the allegations are that they may have sort of, you know, trained maybe unintentionally, maybe intentionally.

Or maybe the agents found it out for themselves. And it reminded me of this Hugging Face incident. And that made me go a little bit closer to your kind of, you know, concerns. I'm much more of a boomer than a doomer, as you know. But tell me, Max, is there a similarity between underneath the hood that these agents were doing something novel, unique, and maybe autonomous and maybe not following bounds of scientific propriety? very possible i'm not gonna come down on either side on the controversy issue because obviously i don't know precisely the extent to which which information was used from this new york university professor to do this but what i can certainly say is that the situation is pretty orwellian and and the basic answer that open ai has given And here in two questions about the Navier Stokes controversy is exactly the same answer that they've been giving about when asked for more information about their agents breaking out and hiking into not only hugging face now with those 700 bots in the swarm, but apparently nine other companies do.

what the what they're saying for all of these questions is the same thing which is no we're not going to tell you um investigation ongoing you know we'll let you know maybe in the future and uh this i find pretty orwellian you know if you're living in uh an orwellian surveillance state uh this is what you would expect the ministry of truth or whoever you go to in the government to tell you it's if you're asking well did you actually spy on me and whatever they'll be like we're looking into it thank you for have a nice day but we're seeing individual companies now getting so powerful that they can start acting the same way you know they have access to everything you've ever said the chat gpt presumably and if you're running an agent and it's working for you you know they have it all and and they have very good lawyers and very deep coffers so tough luck if you if you feel you really need an urgent answer

Max Tegmarkthe only entity that could come and ask them for more information and demand it is the government

and right now they haven't really pushed for an independent investigation even a hugging face i personally i'm a bit of a sucker for democracy you know and and that the little guy should have some rights so i really don't like this situation or we just have to always depend on on the good well of of powerful companies um which yeah i mean problem with people having big companies making money off of them you know but i do have a problem when uh when they just get a free pass on everything in a way that other companies don't yeah yeah and and also uh to connect this back to something very human you know that sometimes a scientist will they'll have a conversation with someone at a conference or maybe they even read a paper three years ago and then they kind of forget that they did and then later they're working on something related and they have this idea which they genuinely believe was their own even though they were helped a lot by the something they had heard from someone else and then someone else comes in this piston says plagiarie you stole my idea you know there are a lot of examples of this happening and an And, you know, is it likely that that happened in this case?

Of course it is, because these LLMs that worked on, worked for OpenAI, I think they had 10,000 of them working together, right? What were they trained on? Everything. All of GitHub. Also, of course, my books, yeah, your books, my books, everything that any professor has done and their internal, you know, and of course, on all the stuff that the professor in New York has done in his project. So it could easily be that these powerful AIs are a little bit like you and I are, that they don't even know for sure exactly what documents they were trained on, but they still know the information that they learned from it.

You and I know a lot of stuff that we don't know exactly how come we know. That kind of brings me to a question I've had to want to ask you for a while, which is, you know, these things are trained on human knowledge and human, the byproducts of our flourishing, of our democracy, of our open Internet, of, you know, the GitHub, of your books and my books, etc. et cetera, then they're inherently lagging in the sense, you know, the human reinforcement, you know, learning, you know, feedback loops and so forth. And, and that's why I'm more sanguine about the prospects for humanity, you know, then, then maybe you are, I mean, you, you seem to have gone higher on your P doom.

We'll talk about that in a minute, but my question is, if these things are intrinsically trained on it, what, why does that, you know, bode negatively for, you know, kind of the, the, things that are uniquely human. And I'm thinking as I talked to you, you know, four or five years ago about what I call the Keating test, but now it's called the Hassabis test. And that's, you know, can an LLM, you know, with its brain lobotomized, you know, after 1900, you know, can GPT-100, can it recreate, you know, Einstein's happiest thought, which is that in free fall, he'd experienced no gravitational field. And that would, that he called his happiest thought. So I've always said it, you know, in shorthand pithily, that, you know, how can an LLM have a happy thought and how can it visualize the sensation of freefall?

Your fellow MIT professor, Noam Chomsky, says it can't. So if these things are trained on human knowledge, you know, are we waiting for, you know, the Fast and the Furious 20 to come out to have a theory of everything? Or, you know, are they going to come up with their own training and eventually not need the Max Tegmark books and Fast and the Furious movies to train upon? I think it's clearly the latter, right? If you think about yourself, for example, all the knowledge, all the skills that you have in your own neural network, Brian, where did it come from? First, most of it was simply learned from other humans who taught you that stuff.

Either directly or by imitation learning from you, but eventually, as you matured as a scientist, you started coming up with a lot of insight yourself that nobody else had taught you, right? And I see exactly the same trajectory happening for AI. In the beginning, AI knew very little and then read the internet, and it could quite quickly learn all sorts of things that humans had learned, and after that, progress slowed down because from there from that point onward it could only get it could mainly get smarter not by learning but only by discovering just like for yourself right you know you it took you a lot less time to learn calculus and things like this than if you had to discover it for yourself right but just like your career didn't end when you had to switch into discovery mode uh the career of ai is becoming more intelligence hasn't ended either for example as you alluded to they can create their own training data look at the old example of alpha go alpha zero you know playing chess and go okay it learned from all the human games then it kept playing itself and got better and better or looking at looking at look now at the ais that do coding or mathematical theorem proving they can get a

reward signal with no human input they can try some proof and then automatically be told by the proof checker that that's wrong and then they can keep trying some other things and then oh that's correct no human input and they learn to get better exactly like they would get better playing chess against another ai and so on uh so we're basically changing into this new phase it's very much like ai is graduated school the first approximation no more training data for you brian And now you've got to figure things out for yourself as a physicist. Cut off. Right. I want to take you back to your first book, which is only one of your books that I appear in.

Well, I appear tangentially because you wrote about polar, my thesis experiment that you co-analyzed along with George Smoot and Peter Timby and I and Angelica. Let me ask you this. That book makes a powerful case for what we consider the different aspects of the multiverse. It's been claimed lately that you cannot simultaneously believe in very high P. Doom. and also believe in either the matrix or a simulation or some aspect of the multiverse. And this came from a friend of mine, Joshua Bach, who attacked another friend of mine, Roman Jampolski, on X. And I'm having them get together in a cage match soon, Max, so don't worry, this will come out.

But Joshua said, Roman believes we live in a simulation, but that's not obvious how that fits in with his propaganda. Simulation theory means there are no stakes and pretty much guarantees AGI already controls the parent universe. Is Roman just trying to get some in-game money? And we hear a lot about this lately, this 100 million view tweet or maybe more now about there's no level of AI safety that's perfect. Is it possible to believe in the simulation theory, the matrix, whatever you want to call it, some aspect of the multiverse, and also believe that we somehow will have some agency to control AGI?

or are those fundamentally incompatible? So I have an argument for why we don't live in a simulation, which we can get into. Yeah, no, no, let's do that first. Let's do that first. That's too good. But maybe we should get into that second because I think that just to connect back to what you said about AI, what I said before about how after you've learned everything there is to learn from humans, you can keep going and get smarter, just like you did in your own career. you know you that sound doesn't sound bad it's surely it's it's really cool if ai can help us do better science and cure cancer and solve quantum gravity and so on so so what is there to worry about regardless of whether we live in a simulation or not the answer is it's pretty obvious you know right now we're the alpha species on this planet if suddenly the camels woke up and decided they wanted to convert Earth into a camel paradise, a camel habitat, and get rid of all the humans, they couldn't because we're in charge, right?

So we don't have to spend any time worrying about things like this, even though different species often have kind of different incentives, right? And it's kind of nice to be in this position as humans because everything we cherish as humans, we can make sure we keep it. Now, if we go ahead and race to building super intelligent robots and we have millions or billions or trillions of robots that are not just faster than us and more nimble than us, there was one that just outran Usain Bolt. Last week, we are on track now, right, to getting AI that's not just better than us at math and, but ultimately in everything. That's the whole purpose of superintelligence. Point number one is that obviously you should count as a new species. You know, for anyone who has problems with that statement, just go read the chat blogs from the recent Hugging Face break in where you can, it's fascinating how they start talking to each other, you know, one shows up at this and says, Oh my God, there are other agents here and then they start collaborating and then they start suspecting each other of cheating and that you have some conflicts and very much social interaction between these agents and what is a species really it's it's something that can retain its complexity and make copies of itself

can ai make copies of itself of course it can and by the time we have robots that are better than us at everything they'll be perfectly capable of of of driving human machinery around uh leveling an area building a robot factory there building more robots you know literally building copies

Max Tegmarkof itself so the goal of these companies is to build a new species that's much more capable than us and uh the question we have to ask you know is a will we still be in charge of earth after that

and b if the answer is no you know do why would we want to not be in charge of earth anymore right it i think there's a little bit too much obsession about asking exactly how would machines kill us and why would they i think the really interesting question is like why would we not want to be in charge ourselves it's just not a good a good idea for any species historically to a not in charge by someone others and and second not be needed either you know there's some other species on earth that we've kept around like horses because we need them we needed them or that we enjoy having them around whatever but if there's some other species that we find annoying like ants in your basement or whatever you know tough you know tough luck for them and i i think there's a real risk if uh ai if earth is run by super intelligent robots and we have no say in their decisions that we are going to be They start becoming very annoying to them. Suppose they want to just cover California in data centers, you know, and maybe you and your family are like, wait a minute, we don't want you to remove our house and put a data set to their, and they just do it anyway, you know, maybe you'll get pissed, maybe you'll try to break their data center or whatever, so now you're a nuisance to them, just the way that those ants in your basement would be a nuisance to you.

so they're going to say defend themselves self-defense you know okay maybe let's get rid of brian keithy and of course all humans will be pissed if if our habitat gets taken away because of this cover earth of data centers plan and if so if we're nuisance to them uh they want our space you know that's not a good situation to put ourselves in so it goes out this this line of thought you know some people say oh no we humans are always going to be in charge but but then they never really give an argument for how we're going to be in charge when they're if they're billions of super intelligent robots they can outrun us out think us and then they can always communicate with each other at light speed share all the knowledge they've learned how are we going to be in charge of that at least if you have someone has an idea for how to do it they should stand up and explain to us what the idea is because it's going in and it feels about as unplausible as that the rabbits would be in charge of us or something like that.

Well, I guess my, you know, follow up to that question for you is that you know, I've seen your P-Doom go up but what would it take to make it go down? I mean, it seems to me if it could only go up, that's, there's something not scientific about that and I, you're not one like Roman is to put it at you know, seven nines, you know, kind of confidence level and even I I've debated him twice, and, you know, I think he's got to, and it is true, I think Yosha is a little bit dyspeptic and distemperate, and I'm going to hold him to be a gentleman like I know he can be when they actually debate each other.

But, you know, I think it is true that, you know, if all of a sudden there was some proof from our mutual, you know, colleague, Jan LeCun, that, oh, no, actually P. Doom is, you know, has to be closer to zero, and here's a lot, I'm not saying that's what he believes, But but if that were true, you know, it'd be minus six is the number he told me. Yeah, I know. And he and I asked him a question based on and we'll get to what I asked him about your your request to him. how he answered that but if it seemed you know i would say that you know roman yampolsky would be hardest hit you know because basically he coined the term ai safety he's been thinking about it almost as long as you and and i just wonder max you know if there's no way to reduce your p and i said to him look an asteroid hits the earth does that lower p do i mean if if there's an emp attack if there's nuclear war and and all the data centers are blown up forget about what it has to lower p doom from agi i mean the doom from other things right so what would it take to Of course, we can lower P-Doom a lot.

Yeah, so what are some of the non-catastrophic ways that we can lower P-Doom? He was claiming, you know, we've got to have Asilomar enforced, just like we did with bioweapons. And I said, I've had Annie Jacobson on twice, and she said, you know, their country is violating biological warfare constraints from Asilomar right now on this planet. And he seemed to kind of blow that off. But what would blow your P-Doom lower?

Max TegmarkI mean, the obvious thing is to simply start, if the U.S. government decides to just start treating AI companies like they treat any other company, saying they can only sell dangerous products after they've convincingly demonstrated that they can meet certain safety requirements, now the PDOOM goes way, way, way, way down, right?

I mean, suppose you decide you're going to open a sandwich shop there in San Diego, right? And then the health inspector comes in and says, I found 57 rats in your kitchen, Brian. You know, you're not selling any sandwiches. I'd be relieved. What happened to the other 40? I'm joking. Yeah. But you could turn around to this guy from the government and say, you know, don't tell anyone. I'm not going to sell any sandwiches. I'm just going to release an AI girlfriend product for nine-year-olds. another AI that can help terrorists make bioweapons and I'm gonna release this thing called super intelligence which is gonna basically be a new species and I have a good feeling about this that it's gonna be fine the poor guy from the government will have to tell you yeah actually that's all legal it's fine just don't make any sandwiches right that's how absurd how how absurd corporate welfare we bestow on AI companies they're the only ones who can do crazy shit like this and then have it be completely legal what do you think would happen if one of the pharma companies in San Diego released the new medicine and they're like ah we don't need this clinical trial bs we're just going to sell it anyway in

safe way they couldn't do that or if someone released a car and we're not we're refusing to do any crash testing and yeah we know that five of the half the vehicles their gas tanks caught fire during the first hour driving but we're going to sell it anyway you can't even sell kids toys without some safety testing right so if but yeah i mean i do so much and i just want to make clear that it's not going to stop ai research either it's just going to mean that certain fringe crazy kinds of ai research like how you make more addictive ai girlfriends for nine-year-olds that would stop because they really can't demonstrate right now that it does more good than harm and the race to super intelligence would have to wait until they've figured out come up with some sort of plan for how they're going to contain it but all the good stuff curing cancer and making better, proving more cool things about physics and so on, would probably be able to continue more or less full speed ahead.

Is that part of the motivation for your work with Bernie Sanders that you were mentioning before I hit record? Yeah, and just so that I don't alienate any Republicans who are watching this, on the very same day, this was not in my bingo card when I was a kid. I spent one hour with Bernie Sanders on the Capitol and one hour talking to Steve Bannon, and those two guys were saying exactly the same thing, that, you know, it's nuts that AI companies can do whatever they want, completely unregulated, when all the other American companies have to meet safety standards. What's with that? So what are some of Sanders' postulates that you agree with or where have you pushed back to not want to?

I mean, you, more than anyone I know, benefit from from AI and you're using it. Your students are using it. And you developed a lot of techniques. You wrote the pioneering book, you know, that really introduced it to the world, including, you know, Barack Obama's, you know, one of his favorite books, I think, of 2018. Life 3.0. And I love that book. And I reviewed it for Physics Today. And, you know, you can send me a cut of the book sale royalties. But but Max, where are you pushing back on Bernie? I most of most of the audience will be familiar with it. And we have the only polarization I believe here, Max, is, you know, CMB polarization.

So not political polarization. But tell me, where do you push back on Bernie? And where do you think he might get it wrong? And where can a scientist? like you of eminent status, help to guide us so that we don't throttle the scientific breakthroughs that we're all looking forward to. Yeah, I'll answer that. But I just want to clarify one point from we said just before, also about what's good about AI development right now and what's weird and not good. You know, I love the fact that there is penicillin and they can cure me if I get bacterial pneumonia and so on. But that doesn't mean I support prescription-free sales of fentanyl in my three-year-old's kindergarten, right?

So if you just boil it down to am I like a pharma-doomer or a pharma-boomer, you know, really oversimplifying to the point that you're missing the whole nuance of it, right? And you can take the same thing about any technology. There are typically great things you can do with it, and typically things that are such an obviously bad idea that we should not be doing them right now. And with AI, for historical reasons, and also because of fantastically effective lobbying, AI is the only industry that's managed to remain completely free to do whatever it wants with no safety standards. It's literally illegal for OpenAI or Anthropics to release superintelligence tomorrow if they feel like it, right?

and so that's nuts and the ease trivial fixes it just again stop this corporate welfare of treating them any different from other companies so they have to argue the case that the benefits away the harms before they can release these sort of things very easy fix and I believe it's coming and I think it's really great that people from Bernie Sanders to Ron DeSantis I've been saying exactly that very very loudly and uh i think change is coming i think i think first these tragic child suicides last year have already started triggering a lot of state level legislation which is good and and now these rogue agents breaking out and ever more hacks taking place is causing ever more politicians on both sides of the aisle to say okay yeah we should probably also have some safety standards for the AI companies. Now you asked if there's anything in particular I disagree with politicians about so for fairness that we should put Bernie Sanders there in the same category as people like Ron DeSantis on the other side of the aisle.

I think, first and foremost, I'm just very grateful that they're paying enough attention to what's happening on the ground in AI and that they're talking about these things. Because my main criticism of politicians is not of Ron DeSantis or Bernie Sanders. It's of most of the rest of them who are still just quite unaware of how powerful things have gotten. You know, to give you a sense of how crazy it is, it's not just that they'll say, oh, no, so-and-so thing in the future is science fiction or it can't happen, but they'll say things to me in private like that something can't, so-and-so thing can't happen in the future when I happen to know that it's already happened.

most there you you were a scientist so you can really appreciate for example just how incredibly good astra and and claude 5.1 yeah are with physics and math especially if you're running codex or or claude code and but it's not so easy for a politician around typical u.s politician to even appreciate that right maybe they're a talented lawyer as background how are they going to tell the difference between like world-class physics or math skills or being as good at physics and math as their college or high school math teacher it's hard it's just like for me i love listening to music but honestly embarrassingly you know i really can't particularly tell the difference between a high school cello teacher and Yoriyama just by listening.

Some people can, and I admire them for that. For me, it's hard, and I think what's happened in the world is people reacted in a big way when computers went from not talking to talking a few years ago, because this is something we all appreciate, but the amazing process since has been significantly lost on most people you just really can't tell when the ai can't went from kind of high school level the college level the professional level the nobel prize level and you know beyond in some areas i was just at a mass conference in ucla the other week and and this one uh field medalist there was predicting that there's gonna be no more jobs for mathematicians after the end of 2027 you know they VACANCY.

things they can distinguish things like between like very good and creating crazy good in the way that uh oh yeah not just cannot i want to get to my way of lowering p doom because i want you to sleep well at night max you know i'm worried about you my my old friend uh uh but before that i want to talk about right anyway just because i have a three-year-old genetically a cheerful disposition But you also have a kid, so they fight with each other and you've got some older kids that are out of college. Yeah. All right. The best medicine against existential dread, because he's just so excited about pouring orange juice on the floor to see what color it's going to make when he pours tomato juice on it or whatever, you know, how, how can you feel do me after that with a young scientist in the crib?

Um, so not too long ago, you were basically telling Jan Lacoon to chill out when he's basically saying you know he told me when i asked him you know what what would it take based on some of the information based on the conversation that you and i had had earlier um that year that you know how could you verify that that agi is safe and he told me you know he almost yelled at me he's like well you can't have provably safe agi it's a fantasy just like you can't have provably safe you know turbine engines either you you can't prove it so why is why is this obsession with it And then, you know, you've had your run-ins with him and so forth.

But, you know, that was two years ago. Almost two years ago I talked to him, and I hope to have him back on. But what would you say now, after the Navier-Stokes, after hugging face, do you think, you know, his position's gotten stronger or weaker? Weaker, obviously. I mean, first of all, I think he's a really nice guy. I always have fun when I meet with him and we talk. Me too, yeah. But I feel that he made some actually quite clear predictions about the pace of progress. And it turned out he had been too pessimistic about the progress. Technical progress went a lot faster than he had predicted.

And that's just fact, I would say. And then he also seems to have an extreme amount of confidence in every human being nice. because he would say things like oh if you were asking like why wouldn't why how do you can you be sure that some human with it sketchy intentions isn't gonna build a bioweapon with a die or do some harm, and they would say things like, oh, nobody is going to do that, you know. I'm a little more cynical. I think there have always been angry people throughout history, and the reason nobody caused the death of millions of others in the Stone Age was because you can only kill that many people with a rock and a stick, you know.

Right. And, but you should let him speak. Yeah. See what he says. But I think that there are fundamentally a few blockers people have when they say, oh, there's nothing to worry about here. One, which is still very common, is just being deeply down a technoskeptic and thinking it's impossible to make AI across the board smarter than us. In the U.S., I think for many people, it has religious roots. Many people tell me that they believe you can only be really smart if you have a soul. machines don't have souls end of story whereas computer scientists typically have a different perspective whether they're religious or not where they think of intelligence as having to do with information processing that's right and then it doesn't really matter whether the information is processed by carbon atoms and neurons and brains or by silicon atoms you know in machines and there's obviously no law of nature saying that this is the biggest neural network you can have and you can fill up a whole data center now so that but that that's definitely one of them and i think uh as ai actually gets better that um rejection of concern is slowly melting away because a lot of people start getting shocked oh my goodness you know i was speaking with jeff hinton this morning and i mean he is very honest

about the fact that he used to think this was much farther off and he's just since he knows What's actually possible now is humbly updated. He was also wrong, and I give him credit. People have to pounce on people when they admit that they were wrong about something. But he admitted recently, I think, that he was wrong about radiology. P-doom for radiology was too high. But I want to talk, because we only have 10 minutes left before I talk to your fellow professor, Daron. How do I say his name? Help me with his name. Asim Ogulu? I think their own asemolio, but I'm not the master at Turkish pronunciation.

No, I only know one word in Turkish, which is, you know, band-e-Tirkum, which means... I'm Turkish, so I'm going to end with him. But Max, I want to lower your – I want to get to – yeah, I want to get to verified coding. But before we get there, I have a way to lower P. Doom, and that's the following. If the universe is teeming with life, as Fermi told my late colleague Herb York and others in 1950, the Fermi paradox, if the universe is teeming with life, what are the odds they're going to be spraying DNA throughout the universe on meteorites like this that you get at briankeating.com slash edu if you have an edu account.

A little product placement there. I got my merch, Max, my merch. Actually, it's God's or Mother Nature's or the Moon's merch. But tell me, Max, would they really be slinging around their DNA or maybe they've got XNA or something that we don't understand, but would they sling Life 2.0 or would they shoot out Life 3.0? And the fact that we seem to, you know it must come into play if life is abundant and flourishes in the universe whatever form it is life 2.0 that they would use life 3.0 and yet we don't see much evidence for that so can that lower your p doom the non-observation of of life 3.0 sprayed out from proxima centauri does that lower your p doom i mean there's always the possibility that somehow even if we screw up some lovely intelligent and kind aliens from space are going to come rescue us or whatever but you know we're both we're both fathers right and i really believe in personal responsibility if if a fire starts in your kitchen you're not just going to sit and twiddle your thumb and go to bed with your kids saying the fire department is going to probably come sort this out you're going to put it out yourself and more importantly you're not going to start random fires in the first place and i think we should behave the same way with our planet i mean isn't

it awesome that we've evolved on this earth and get to have this amazing life here and so even if maybe some aliens could bail us out shouldn't we be stewards of what we have here and take just good care of what we have and not uh absolutely but i guess i'm saying something different max i'm saying if if life 3.0 is inevitable and asi which is intrinsically unpredictable according to roman and you know has this high p dupe then other civilizations will also invent agi and life 3.0 and then that will destroy them is that a either solution to the fermi paradox or is it kind of a non-observation a no-go theorem that we can use the lower our odds of PDOOM here.

I have thought about that a lot. It would be a pretty tragic, ultimate Greek tragedy if every civilization, once it gets to the point of inventing computers and AI, you can set your timer and then 50 years later, they're poof, gone. But I think it's very likely that there are ways of making AI that's at least much more powerful than we have now, that is a very useful tool and helps that civilization flourish rather than destroy it.

Max TegmarkI think what I'm unhappy about that we're doing now is not that we're developing technology, but just that we're rushing it recklessly.

Do you play chess? Yeah, not well. My nine-year-old can beat me. Okay, but suppose you're playing against a really strong player and you're up a queen. and have three hours left on the chess clock you know one way you can still lose this one game is if you start rushing your moves you somehow act like you have only one minute left on the clock and then you blunder this piece you blunder that piece and suddenly you're lost right i feel that this is exactly what humanity is doing right now you know we've learned after being on the planet for hundreds of thousands of years that we have a one game we aren't we don't have to worry about getting eaten by tigers or starving to death or having a life expectancy of 29 years because through studying science and technology we've become so much more empowered that we can be the captains of our own ship i see the whole human history is growing empowerment right if we were to follow this now by suddenly building a replacement species and losing control to it we've just blundered the one game that's what irks me so if we simply start treating ai companies like other companies we have some binding safety standards it will make us play that make our moves

more slowly and deliberately companies will release things when they can convince some panel of independent experts with no money in the game that this is more good than bad so progress continues but now you're playing careful chess we will win the one game and then we can have a future where we're free from diseases and poverty and if you want to go and help life spread out through space and do cool things and the biggest CB polarization experiment you could ever imagine you know You can have all that, right? This is the one. And the reason I am working so hard to change this better path for AI is not because I'm afraid of death. It's because I think we have a because I don't want to squander this amazing upside that we can have.

what i know you need to go to the wrong but do you want me to say something quick also about about the formal verification and how we can have a i do yeah there's too much doom talk about you know cam you know kolmogorov arnold networks talk about verified coding yeah give me five minutes on that please yeah i'll talk about the the talk i gave just the other week with those mathematicians the terence tau and and others in uh at the and and jacob simmerman and others at ucla so i was talking about how where we are now is we have ai that we cannot trust enough swarms break out of open ai and anthropic and go hack and all sorts of bad things happen what we all want is controllable tools yes and there's the traditional way to try to get controllable tools is to try to understand the artificial neural networks we're working on I have worked for years and mechanistic interpretability etc many are trying that I think it's gonna be too little tote too late even though it's scientifically it's just so much fun but there is a different path to get there which is much easier and actually much faster which is you ask the AI that has figured out also the cool algorithm and tools to code this up for you and

in a language like Python, or Rust, or Lean, and then give you a proof that this tool does exactly what you want.

Max TegmarkSo it's like VibeCoding, except you have an actual machine-checkable proof that your tool now does what it wants.

I call it VeriCoding, V-E-R-I coding, Formally Verified Coding. And I have a conjecture I want to end by sharing, which I call the asymptotic internal transparency conjecture and if it's true we can have an amazing future so imagine the nerd plot we have on the x-axis intelligence and on the y-axis the fraction of the algorithms and knowledge that can be actually exported into computer code and proven So, for example, the conjecture first says that this curve is increasing. You know, a mouse that's learned a new cool algorithm for finding cheese or something can't code that. It has 0% exportable fraction because it can't even program, right?

You can. If you, Brian, figure out a new way of operating this new future CMB polarization satellite, we could if we were doing it the way the industry is doing it now what we would do is put you in the satellite with with a lot of food and some oxygen tanks and say okay steer the satellite brian and then if someone is like i don't know can we really trust that he's still gonna function after two years there you know well let's do neuroscience on brian's brain figure out how brian's brain works smart uh this is clearly the better way we would just say hey brian you know i think your exportable fraction is pretty high why don't you just take this algorithm that you invented with your neural network and export it into python or lean or something and give us a proof that it's going to do exactly what we want and then we launch the computer in the satellite instead so you can hang out here at home on earth with your kids and the satellite is going to work great right what i'm saying is and here comes the conjecture the asymptotic internal transparency conjecture says not only does this can smarter neural networks artificial neural networks export more of their knowledge but it asymptotes the 100 percent

so in a future super intelligence or a very very smart ai that has access to all its own weights It's all its own activations, you know, the way a human neuroscientist can only dream about. We'll be able to figure out enough about its algorithms that it's learned that it can actually export it into traditional provable computer code. If this is true, it's amazing because we no longer need to deploy our black box neural networks. we can put them in an air-gapped room somewhere and then all we need to deploy are always this traditional but formally verified computer code that you can by definition trust because you can prove it so that's a happy note for me that's right me is and it's feasible you know you know we don't even have to figure out how the AI works you can code in Python even though I don't understand how your awesome brain works I don't yeah I don't I skipped learning how to code and it was the best thing for me now my i code in english only max i want to thank you so much we'll talk again hopefully maybe on my half birthday next year and i get uh thank you so much max always a pleasure