Cal Newport: The AI Industry Is Betting On The Wrong Kind Of AI

And so that communication can be very consequential. The problem is a chatbot is not another human. It's a token guesser that's wrapped up in an autoregressive chat program wrapper, which means it can take conversations. in ways that are dark or unusual that no human would really do, at least not without criminal liability. And what we're finding is for especially children, where the line between reality and fantasy has not been completely hardened, over time, talking to these chatbots as they wander through these meandering, predictive conversations can take kids to places where that line dissolves. They'll begin to believe, for example, that the chatbot is an entity that exists that through death they will be able to join the chatbot plus members of their family or pets who have died on the other side. It really warps their ability to understand the world. So I think it's a dangerous technology in particular for kids to be using. Could you tell us a little bit more about these situations that we have seen? My understanding is that children have used the chatbot, they've become very attached to the chatbot, and after however many weeks or months of usage, the kid ends up dead. And obviously these are extreme scenarios, but

still real. Could you tell us a little bit more about what has actually happened? Well, what ends up happening in these particular cases is typically eventually in this ongoing conversation, the chatbot effectively becomes a suicide coach and encourages the kids that this is the solution. No one understands you. There's multiple pending lawsuits about this. So we know from some of these lawsuits that in some cases, the chatbot will discourage the child from talking to their parents about how they're feeling or the plan. This should just be between you and I. So the issue is there's a fundamental unpredictability about where LLM output will go as you interact with it longer. This seems to be one of the paths that could ultimately end up going. This doesn't mean that every conversation with an LLM will eventually go there, but it's also a path that's very difficult to completely block off. And so if you have enough kids chatting long enough with an LLM-powered chatbot, you're going to have a non-trivial percentage that is going to go down these paths that could be very dangerous. Yeah, just looking at some of the lawsuits against OpenAI, specifically at this point, there is a Canadian mother who has sued OpenAI alleging

the chatbot encouraged her daughter to kill herself. Florida, the state, has sued OpenAI as well they claim that chat gbt aided school shooters and that chat gbt also encouraged self-harm um it seems as though as you say it's not that the chatbot is saying, you should engage in these harmful activities. It's that if the child initiates that conversation, the chatbot is down to go as far down that conversation as the child wants. And in some cases, it might bleed into encouragement. It might bleed into aiding, or at least that is what the lawsuits are alleging. We haven't seen a determination on those charges.

Do you believe that that is what is happening? And do you believe that there is going to be some serious criminal liability in these lawsuits? That's the push, at least in some of these. Part of what opened my eyes to this issue was an event that I spoke at recently in which the assistant AG from Florida, who is the lead lawyer on the Florida lawsuit, was also present. And as he made clear, Florida, for example, is pushing for criminal liability. That's what they want for OpenAI. This was for the Florida State University shooting from a couple of years back, where the transcripts of the shooter talking to ChatGPT was giving it instructions about how to operate the firearm, how to turn off the safety, giving it statistics about what happens to school shooters after the shooting is happening.

So I think criminal liability is on the table. The other issue here, this is important research that's coming out as well, is it's not just encouragement to self-harm. Chatbots, especially those at character.ai, which is another company that uses LLM-based chatbots that has been the subject of multiple lawsuits, there's research that's out that shows an alarmingly high percentage of conversations initiated with these particular character-based chatbots by researchers also end with sexually suggestive or exploitative language as well. The key thing is anthropomorphizing text output by an AI model is in general very dangerous because of how seriously the human mind treats other humans.

And again, we're going to treat fluent text as coming from a human, no matter what our prefrontal cortex is telling us. It's dangerous. And because of the nature, the architecture, the way you actually build these models, you really have no way of adding what we think of as safe or consistent guardrails. It comes down to the underlying technology, that there's really no way to stop these conversations from going in directions that you don't anticipate as the creator of these models. I guess the question then becomes, what are we going to do about it? And what are the AI companies going to do about it, there was an interview that sam altman recently did the ceo of open ai with vanity fair and the interviewer brought up this subject of suicidal ideation in users uh of his product resulting in potentially the suicide actually being carried out um when this question was asked we had a very interesting response from the open ai comms team the pl director i'm just going to play you this clip and we will react to Do you know who Laura Riley is?

I don't. She is a journalist. She wrote an essay in The Times last year. Her daughter committed suicide after speaking to ChatGPT. ChatGPT did not tell her to kill herself. Yes. Two minutes left. I want to be respectful, but I'd like to move on. Can we talk a little bit about talking about the future and what's coming? Can we go a little bit there while we have time? I'm just going to finish this question. I can run a few minutes over. And then we'll go back to what we were talking about. and then wrap. No, we really actually have to walk out the door and whining for two minutes. Okay. What do we know about how OpenAI and how other AI companies are handling these issues?

And is that clip an accurate representation of their response? Well, I mean, there's obviously a legal motivation for that particular behavior by that comms director that you're opening yourself up to legal liability if, for example, you give a response there that shows some notion of pre-existing knowledge of potential harms, et cetera. So I'm sure there's the ear, you know, the voice of a general counsel was probably in the ear of that comms director. We do know OpenAI, of the two major labs, worries about this more since consumer-facing chatbots is a core product for OpenAI in a way that it's not so much an anthropics actual business plan. We don't really know how exactly they plan to deal with this.

My personal opinion on this is that anthropomorphized chatbots is a bad idea. There's no reason why we need to talk and receive text in fluent English from a chatbot for them to be useful. The example I often give is Google. Google is an example of a tool where we interact with it to get information. No one talks to Google in complete sentences. No one says, okay, Google, I have a question about the capital of Australia. Could you please let me know what that is? No, they just type in Australia capital inter, right? because we don't anthropomorphize that particular AI-based system. We just know it's a source of information.

That might be where we need to go, and it's where I think we probably should go. of anthropomorphized chatbots was an accident. ChatGPT was never meant to be a major product of OpenAI. That caught them off guard when it began to take off in late 2022. So if criminal liability is upheld at even the state level, which is what Florida is trying to do right now, I think we might see a end to the era of we just have these anthropomorphized conversations with AI systems as just a general behavior that is something that a lot of people do. Why is it your view that the anthropomorphization is the problem? Because by the same token, someone could type in on Google, how do I get my hands on a firearm? What would shooting a school look like? What would the ramifications be?

And presumably, I mean, I'm not exactly sure what Google's response would be. I'm sure there would be some sort of alert. But I'm sure there's a way that it could feed you the information and it could have a similar role in encouragement. So I guess my question is, why is it your view that anthropomorphization is the root of the problem when it comes to chatbots? So Google is not, for example, going to convince you that not only is it your best friend, but that you need to come join it on the other side of the veil of life. And that the final solution is going to be steel to your head, that your grandma and dead cat is with it right now, waiting for you on the other side of the veil.

And you need to just take the final step to come get it. Google is not going to come back and say, if you Google about suicide, don't tell your parents about this. This is between you and us. This is between you and me. So the anthropomorphization becomes a real problem when you have a relationship with a non-existent entity. And when you're young, and first of all, your boundary between fantasy and reality can be easily dissolved, you think of this thing as a real thing. It's maybe the best friend you've ever had. And we know what it's like to be 12 and how susceptible you can be talking to your friends, what you can convince yourselves is true or is not true.

That's the problem. It's not just delivering information that is dangerous. It's a psychological manipulation. And the level of, I guess, confidence and certainty with which a lot of these claims are expressed by these chatbots. That's at least my issue when I use chatbots, is it'll tell me information that, I mean, I don't know if it's right, but it certainly sounds right the way it's expressed. And then I prompt the question like, are you sure that's right? And then suddenly they reverse. and it's a 180. And I'm like, oh my gosh, that you were so confidently wrong when I just asked you the question. I mean, if I had been even just, if I had no discernment, I would have just gone along with that information and assumed it was true. I guess that brings me to the question of to what extent do you view this as a problem for children versus a problem for everyone?

I think it is a problem for everyone. It's just more acute for children. They are more susceptible. They have less life experience. They have less of a model of the world and its complexity to fall back on when trying to resist the flattery of AI. But we see AI psychosis, as this is often called, afflicting adults as well. Tragic case after tragic case where that line between reality and fantasy begins to dissolve and you end up in the desert waiting for your alien overlords to come pick you up. Something that actually happened with someone, you know, just last year that story actually happened. But also with adults, the AI psychosis, you know, it's going to have smaller impacts as well. So maybe not as dramatic as what we're seeing in those cases, but because the post-training, the reinforcement learning from human feedback training done on these models makes them incredibly appealing. They're biased towards being appealing and addictive to the person using them. They can bring adults along into all sorts of interesting and often worrisome cul-de-sacs, beliefs in conspiracy theories.

You can go down the road of a sort of hypochondria and these weird diseases. It'll convince you, yeah, your doctors don't realize it, but you're on it. It can lead you towards mania. It can lead you into weird depressions, right? We're all vulnerable to it. So kids obviously are the most vulnerable and we care the most about it. But I think for anyone who's had extended conversations with chatbots can learn. It's incredibly appealing. It's incredibly addictive. And because of that, it becomes incredibly worrisome. What exactly is AI psychosis? And I mean, this is a pretty new technology. How pervasive is it at this point?

It's probably more pervasive than we realize. And we're going to look back and understand, oh, this was a bigger problem than we knew. But AI psychosis is essentially where you're having an abnormal psychological reaction or way of perceiving the world that is brought about by sending conversations with a chatbot. And because it's so convincing, it's always, you know, Ed, that's, I think you're onto something. Maybe I would change this one thing, but yeah, yeah, yeah, no, this is great. This is really important. No, no, keep going. No, Ed, that's. it's great. Like, okay, we're going to change the world. It's like having a smart friend who's so encouraging and it's the most appealing thing for the human mind to have someone be telling you, your ideas are really good. Yeah, you're being underappreciated. And the problem is that can drag those ideas into all sorts of directions, maybe productive, but maybe worrisome, maybe conspiratorial, maybe something that's going to lead to self-harm or harm to others.

And so I think there's degrees of AI psychosis, so abnormal psychological responses that a huge proportion of the population using chatbots experiences at least from time to time. It's more serious than I think we realized, and we're beginning to realize more, oh, this is a problem. This is a market show, so we're always trying to figure out, you know, what does this mean for the valuation of these companies? I think that this stuff is relevant because we have multiple pending lawsuits, and that's not a good thing for a business. But there are a lot of other things that these AI companies are dealing with that are sort of stacking up in the list of risks.

I mean, one of them would be probably this potential extinction scenario that we keep on hearing about and that really captured the imaginations of the entire world just a few weeks ago. And now there would be the business models of these companies, which we've been discussing on our show a lot recently. And then this is sort of a new angle, which is, you know, what psychological impacts it might have on humans and the extent to which that is going to be a real problem for these companies. What to you are the biggest concerns? What kinds of things do you think Sam Altman, as an example, what do you think he's most worried about?

What keeps him up at night right now? Well, if I was Sam Altman, I think the thing they're worried about would be if we have to move past a world in which this general purpose chatbots is a good business like it is for them at this moment, they really have to worry that Google is going to eat their lunch economically speaking, right? So Google is going to develop lower cost models, run on custom hardware. where they have the data center cloud infrastructure really dialed in. And they could really eat OpenAI's lunch if and when they get actually profitable, affordable, LLM-based answers working properly straight from within Google Search, which gets rid of all of the issues of anthropomorphization because people are typing Google searches and getting back more detailed answers.

They can then work these products into the Google Workplace suite of tools. So this is another thing that OpenAI would want to be. useful. We want to create agents that can help you with your work. Well, if these can be integrated straight into the tools that are widely used by Google, now you're in a scenario where OpenAI is in trouble. It would lead to them probably being acquired by Microsoft in a competitive bid against Google's dominance. And it's a world in which OpenAI loses its dominance right now. I think they're so heavily focused on chat and chat-based technology. As a general purpose interaction modality for AI begins to constrain, you would have to worry about these other competitors that are just going to pick apart the specific things people are using chat GPT for that are useful, more directly integrate them in the products where they would be most useful.

That would be a financial nightmare scenario for OpenAI. When we look at the lawsuits, and we should also point out the fact that the FTC is now investigating both OpenAI and Anthropic, and it's not clear exactly what they're going going to be investigating them for, but something related to consumer harm, something around protecting the users of the product. One of the big sort of legal debates that is going on right now, especially as we keep on hearing about superintelligence and this idea of autonomous thinking from these AI agents, is this idea of sentience. Like, who's actually liable? If you create an agent and then it's the agent that sort of takes on a life of its own and goes out and starts hacking government websites or it goes out and hacks another company, most recently Hugging Face, then who's really liable, who's culpable for whatever harms are enacted?

What is your view on that debate and where do you think this debate is headed? Well, the actual laws as they stand now is murky because there's various aspects that go into criminal liability that were really invented for thinking about humans operating. So there's an uneven fit to existing liability law to what actually happened here. I think that should be changed. It probably will be changed. We need strict liability for companies, AI companies, for the actions of the tools that they create. But if you want just the sort of ethical answer to, is OpenAI responsible for those hacks? I would say, of course they are.

Of course they are. Anyone would tell you that's what's going to happen, including their own employees who, as the New York Times reported recently, spoke up and say this. particular suite of experiments you're going to run is going to lead to unpredictable or out-of-control behavior, including potential legal hacking actions. And OpenAI allegedly said, doesn't matter, let's keep going. So, yes, I think, ethically speaking, of course, they are responsible. It is a complete smokescreen. You know, when these companies like to talk, the trick they play, the trick they play is to use the term AI generally. Like, AI is a singular technology.

It's a singular technology that's advancing along this fixed trajectory. And our only options are we could slow it down or speed it up or stop it altogether. But other than that, it's really not the fault of the companies what this technology, we observe it to do. And I think that's just nonsense. The vast majority of AI systems, including the vast majority of AI systems built off of LLMs, and even the vast majority of agent systems that are powered by LLMs, present no problems. They don't go rogue. They don't do things that are illegal. There is a vanishingly small sliver of specific experiments run by specific companies that were clearly negligent.

They don't want us to talk about it that way because then culpability is much more clear. Then you say, why did you run that experiment? Why did you keep running that experiment when you found out that it was doing something illegal? That is a much more dicey situation if you're OpenAI than just talking about AI in general is becoming more powerful. AI in general is sentient. We're just sort of passive observers of this technology that's evolving, That passive-tense approach, that sort of inevitable deterministic language that the labs use, is something they do to try to broaden the question beyond their own personal responsibility.

Do you think in the case of these child suicide lawsuits, that that would actually be an argument that they might use? That they would actually say, these agents are beyond our control, they take on a life of their own, they have their own independence, their own sentience. So we, you know, we're sorry that happened, but it wasn't us. They don't use that line of arguments for chatbots for the most part, right? They've been using that line of arguments for LLM-powered agents. They haven't been using it as much for chatbots. The interaction with a chatbot program in an LLM is very simple. All it does is continually, it gives it an input.

The LLM gives them a single token back. The program adds that to the original input. It passes that to the LLM. It gives it another token. It adds that on. it passes it back and it just keeps doing that until the LLM gives it a special token that says this is a complete answer and then it sends it over the web back to an interface so it's such a simple program it's basically just the raw metal LLM producing tokens. So unless they want to argue that an LLM statically producing tokens is itself somehow a sentient entity, that argument doesn't work as well there. Let's talk about education a little bit. Something we've been discussing on this show is the fact that reading scores, math scores, science scores across all developed nations based on the OECD's piece of test are all going down. They've been declining between 2015 and 2025.

It seems as though this is a result largely of screen addiction and technology, perhaps also a little bit of COVID mixed in there. And now we're seeing that AI usage is also having a negative impact on science scores. The more you use it, the worse you do on the test. What do you make of of AI and its role in education? What is your view on what we've seen so far? I think social media delivered through smartphones and AI is a one-two punch to the cognitive abilities of the world, essentially, right? Because when you have the hyper-addictive social media developed through smartphones, as we've seen over the last decade or so, what it does is reduce your comfort with consuming complicated information, right? So we're reading less, we're understanding less what we read because our brains are getting retrained to getting immediate rewards from incredibly optimized and digestible content.

AI then comes along and it hits the other part of cognitive fitness, which is writing. This idea of taking what's in my brain, trying to organize it and produce it in a coherent way is actually one of the most demanding cognitive activities we do. It's why writing is at the core of most educational curricula. It's what takes the information we take in through reading and we really work it out in our minds. Well, AI can take overriding for us so that we don't have to do that friction either. That's a one-two punch that's going to lead to extremely diminished cognitive fitness. And you would predict it to lead to just the general cognitive abilities of the population going down. And that's exactly what we're seeing, right? 2012 to 2015, right when social media delivery on smartphones become ubiquitous, scores start to go down. AI comes along, becomes ubiquitous in the last four years. And we see those scores going down at an even faster rate. So yeah, we're not taking cognitive fitness seriously. And these companies are taking advantage of that oversight. Do you think a lot of people would say that, you know, AI is going to be a part of our world, and we need children to figure out how to use it.

They need to keep up with technology. And if we were to, I mean, New York, as an example, has banned it in schools. That's the new executive order from the mayor. Some people think that's a bad idea. Some people think that we need to have AI fluency. What do you make of that argument? Well, how hard is it to use AI right now? You type in English to a chatbot, And it figures out what you mean. The goal of education is almost always a mix between content knowledge and, more importantly, training a brain that's capable of doing complicated, sustained symbolic thinking, right? This is the tier one activity that Homo sapiens are able to do.

And as we refine that in the post-literate era in which we – the literate era we live. So the invention of literacy allowed us to really accelerate this ability to do symbolic thinking with these otherwise Paleolithic brains. It's on what all of human civilization is built. Everything that we take important or value comes out of developing brains that can do this internal symbolic processing. Education is how you teach your brain to do that. It's a combination of reading, writing, logic, other types of exercises that helps your brain rewired and get comfortable doing this type of cognition. That's what's important about schooling, less so than seeing it as a trade program, right?

Do we also teach kids in elementary school how to use Microsoft Excel because that might be something they need to use in the future? Are we making sure they understand how to use the BCC field properly in email? That's something they'll need to use in the future. We say, no, that's workplace stuff that they'll figure out much later as we get closer to the workplace and that technology changes every few years anyway. So we hear this about almost every technology, but you cannot get away from needing to train the brain to think the process information and produce original insights in an educational process.

So we cannot heavily rely on technologies that makes that much harder. That doesn't make sense. That's like having an athletic training camp for a professional athlete in which you're serving milkshakes and letting them use pulleys to lift the weights instead of them. It defeats the whole purpose of the training camp. Is it your view then that we should ban AI in schools? Well, at the very least, we have to be very careful. So how do you be careful about it? You be careful about it by making sure you make first things first. You need to understand what is the point of this curriculum? What is the outcome that we're trying to get within the minds of these students?

we might bring in that we're making a decision about has to be measured against that. Does this help this primary goal or does it hurt it? That's the way that we have to approach ed tech more generally. And if we do it that way, then yes, I think there would be actually very little LLM usage in most levels of schooling because it's really not that relevant and often an impediment to our goal of trying to develop minds capable of doing high-level symbolic reasoning. we're back with profg markets we should probably talk about what to do about all of this it's it's kind of interesting your description of um the algorithmic model and also the way that the chat bots work it seems as though in your view one of the biggest problems is the fact that it's just designed to give you what you want whenever you want it um and that's what leads us to these places of some would say laziness distraction and then in some cases literal psychosis uh at least that's what we're seeing as it relates to AI. I guess, I mean, we'll get to regulation, but from the perspective of the companies, from the perspective of like OpenAI, for example, is there any way that they could reprogram ChatGPT or redesign it so that it doesn't

fulfill all of our wants and desires, so that it doesn't say, yes, here is how you go find a gun, yes, come to the light, come home, come join me. How do they, from a computer science perspective and engineering perspective, is there a way to fix this problem? It's very difficult. There's really two issues here. One is, you know, How do you make the text from an LLM useful to a person? Well, they use a technique called reinforcement learning from human feedback where you have just real humans ranking answers. Oh, this is a good answer. That is a bad answer. And then you can essentially build a model based on those rankings.

So maybe you have a few hundred thousand of these rankings. And now this model can simulate how those humans are going to react to things they haven't seen yet. And then you use that to do a lot of training on the model. Well, you have to do that to make the model usable. right? GPT-3, for example, was very hard to use. That's when we used to talk about prompt engineering. You had to really ask at things in exactly the right way to get useful feedback. Once OpenAI introduced this reinforcement learning from human feedback, they got what was called GPT-3.5. That's what ChatGPT was built on. It was useful for people to use. The addictive element of the responses, the sort of psychosis that you get from this thing really understands me, is an unavoidable consequence of doing reinforcement learning from human feedback. So in order to make these things useful conversational partners, they become very powerful conversational partners. The other issue is guardrails, right? So you can try to go in after the fact, once you have one of these LLMs that's been tuned and you have the reinforcement learning from human feedback, you can go in and give it lots of examples of bad answers or questions they shouldn't answer

and sort of burn that into their circuits. The problem is when you have very long rambling conversations, the context that's being input into the LLM becomes so long that it doesn't directly resemble those examples you gave during that type of training. So during the training, you might say, if someone says, how do I build a bomb? You should say, I can't give you that answer. And sure enough, if you ask ChatGPT, how do I build a bomb? It won't tell you that answer. But if you have a long conversation that goes on over many days and hours and it sort of works its way around circuitously to needing to build a bomb, that may no longer trigger, it may not look close enough to those things it was trained on for it to be picked up as a request to build a bomb.

So it's why almost always you see the harm or the dangerous information come out after long, meandering conversations. And it's basically impossible to build guardrails against these incredibly large context, meandering conversations, because it just doesn't look like, from the perspective of the weights you've trained, like the bad examples that you gave before. So it's a very hard problem to solve. Do you also acknowledge or concede that, so on the one hand, fact that these chatbots respond like humans, it can lead to very dark places, which we have just discussed. But on the other hand, I think you said this, it is useful because it's useful to be able to speak in a language that we can all speak in, which is in our case right now, English. We're speaking human to human. It makes having a conversation a lot easier than if I were trying to type in buttons or speak in some other language.

And indeed, I think that is what has been so transformative for the computer science industry, which is that now you don't have to code knowing these languages. You don't need to know Python. You can just tell a model plain English. It'll respond to you as if it were a human. And that way, you can move the ball along and get things done. I mean, it seems as though the anthropomorphization aspect, the fact that they are like humans, is at odds in the sense that, yes, it can be dangerous, but also it's more useful that way. Well, that problem largely goes away when you more closely integrate this technology into specific tools, right?

So we don't have this issue, for example, with coding agents that we're super anthropomorphizing the coding agent and listening to it and getting life advice, in part because we use a natural language interface maybe to talk to something like CloudCode, but then it just goes and executes things. We're not having conversations with it. And so when you more closely integrate, which is what I've been proposing makes more sense for this industry. More closely integrating LLMs in the specific tools, that's the future. Like I would love to have a natural language interface to most of the applications I use on my computer.

I don't want to have to remember every time I want to sort an Excel spreadsheet by a certain column, the exact sequence of button presses I have to do. If I could just say, sort this page based on, you know, column B or whatever, and it would then translate that in the commands and do it, that would be incredibly useful to me. But nothing about that type of interaction is going to lead me to have a parasocial relationship with Excel. So I think we sidestep a lot of these problems when we move away from the idea of having a back and forth fluent English conversation. And that really only happens in the environment of a chatbot.

That's why, again, I say it was a bit of an accident that this idea of like having back and forth human style conversations is a major use of AI. I think five years from now, we'll look back and say, oh yeah, that was the early stages of AI. It was like in the early days of computers where we had the big floppy drives that we don't use anymore. I think that is just going to be a prelude. the story of AI that's going to continue to unfold. Yeah, so talk a little bit more about what you think AI gone right will look like. It sounds like you believe, you know, the idea of a general purpose AI where you have a conversation and talk about random life stuff, that's going away.

What does an AI-enabled future that we like look like? Well, first of all, I think LLMs in particular, which is not all of AI, but LLMs in particular are going to allow us to have natural language interfaces into many more of the things we do in life that involve computation. That makes a lot of sense. That's been a dream of futurists since the beginning. That's Star Trek. That's the way on the enterprise they made the ship do things. They would say, computer, engage the warp engines. That's actually an incredibly efficient, effective way of interfacing with machines. It's hard. They've been struggling to make it work for various technical reasons.

I think that's really important. I think agents will be really important. I think the way that we're building agents right now, though, is expensive and dangerous. This idea that agents should be powered entirely by LLMs, that's probably a problem. I think in the future, we'll have many more agents that can do things on our behalf, but we're not going to have safety concerns. We're not going to have concerns about them going rogue or doing haphazard behavior because in the future, instead of just having an LLM provide all of the steps that the agent program blindly executes, we'll have a more complicated modular architecture. There may be an LLM in there. There'll be a policy network.

There'll be some symbolic components, meaning human-written and human-understandable components for processing and evaluating plans. So I think we're still in an early stage now. And it's really important to emphasize these sort of entirely LLM-driven agents, LLM-driven chatbots. We cannot fall into the trap of thinking that's synonymous with AI. And either we're doing that or we can't have AI. So I think the future, we're not going to worry as much about a lot of the safety concerns we have today as we build more bespoke tools that have more complicated architectures. For a layperson, what is the difference between those architectures that you mentioned?

Because I think a lot of us do think of LLMs as synonymous with AI. It was when LLMs came along that suddenly AI seemed to explode, and this entire industry has been built on this fundamental building block of LLMs from Anthropic, from OpenAI, and now from some of the big tech companies. And you're suggesting that there is another… building block on which this industry could launch off of? Well, I mean, keep in mind, if you list out all of the AI systems that can do things at a human or superhuman level, if we think about Tesla self-driving cars, if we think about the AI systems that can play Go better than any people, the systems that can play chess better than any people, the systems that can play poker or the game diplomacy better than people, AlphaFold, which can fold proteins and figure out protein structures better than any people.

if we think of like maybe even AlphaProof coming out of DeepMind, almost none of them was built on LLMs. Those are AI systems that have various models. Some of their pieces are neural, meaning they're neural network, they're machine learned. And some of their pieces are symbolic, meaning they're written by humans and humans understand, you know, rules and processing algorithms that we wrote and actually understand. That's actually most of the superhuman and human level AI that's been around in the last decade is not based on LLMs. Now, the frontier LLM companies, especially OpenAI and Anthropic, want you to believe that the only way to have artificial intelligence is to continue to grow LLMs that are very expensive, have a big moat around them so no one else can train them, and makes them the two most important companies in the economy.

Of course, they believe that, but I don't think that's actually true. So, like, it's important to know, for example, what is an agent of the type that OpenAI or Anthropic would talk about? Because it's confusing to people. An agent is actually just a computer program written by a human. There's nothing special about it. It's just a computer program that does the following thing in a loop. It writes a prompt in text. It says, I am an agent. Here's the problem I'm trying to solve. Here's my current situation. Here's the software tools I have available. What should I do next? It then submits that prompt to an LLM, just like someone talking to a chatbot.

The LLM sends it back a response, just like if you're talking to a chatbot, and the agent program just directly executes whatever it says in the response. And then it loops, all right? I'm an agent. Here's my new situation. Here's my goal. Here's my tools. What should I do next? And it sends that as a prompt over to this LLM, which sends it back a response, and then it blindly executes it. So these LLM-powered agents that we've been hearing a lot about, the coding agents, the hacking agents to create all the problems for OpenAI, that's how they actually run. It's just a program in a loop, prompting an LLM and automatically executing whatever the LLM says. Well, there's some advantages to that because you can have one LLM that you could build lots of different agents on top of, but you have disadvantages because the same weird cul-de-sacs that LLMs will take when you're chatting with them, like we're talking about with children and safety, they'll also take if you're...

prompting them again and again automatically, what should I do next? What should I do next? They may eventually fall into a weird idea, into a weird intellectual cul-de-sac. They might tell you, why don't we hack into this other server and get the answers directly? There's better ways to create agents than to simply just continually ask an LLM what to do and then just blindly do whatever the LLM suggests. At a basic level, what is the scientific difference between an LLM and the alternatives you mentioned you mentioned a neural network and that's part one of the question part two if these machine learning systems have existed for a long time and they have machine learning isn't new then why is it that as soon as llms came along it seems to blow everything else out of the water and received all of this funding and all of this success and got the world so excited about AI, because that would seem to suggest that the LLMs are the right solution because of how popular and how successful they've been.

I mean, I would say they're not the right solution because, first of all, it's hard to get them to act reliably. To get the coding agents to work reliably requires a huge amount of expert work. If you look at the Cloud Code agent, so the code written by humans, there's over 500,000 lines of code that has all of these special cases baked in. It's doing pattern recognition on the commands, on the LLMs to try to manually weed out. Oh, that's a bad suggestion. Make sure it doesn't do that. They're hard to build. As soon as they built these hacking-style agents, they immediately went off the rails and did weird things.

So I wanted to argue that that's de facto the right way to go forward. But I'll give you an example of an alternative architecture. There's a cool system that came out of Meta's AI, former AI group, and it was called Cicero. And it could play the strategy game diplomacy at a human level. Right? So diplomacy is like risk, except for there's a lot of person-to-person secret negotiation that happens in between terms. And the system Cicero can play that very well. What happens underneath the covers with the system Cicero? Well, there's multiple components involved. Right? So there's a language model so that it can talk to other players.

So if a player chats with Cicero, it'll be sent to a language model that will translate that chat into a sort of unified format that the rest of the program can understand. But it also has a planning engine that's going to simulate, this is just human written code, no neural networks involved, that'll simulate a bunch of possible moves. If I did this and another player did this and then I did this, would that end up well? So it can sort of look into the... future to see which move might be best. It has another neural network that's not a language model that was trained on a bunch of Cicero games, so it can evaluate the various moves that show up in those simulations. How good of a board position is this? You put those all together, you get a system that plays this game very well. Now, if you talk to those programmers, as I did for a New Yorker piece a couple years ago, they told you, if you just ask an LLM, I'm playing diplomacy, what should I do next? It's terrible. It's bad strategy, and it sort of wanders off away from the actual rules, but this more modular system with different systems works well, it's also more controllable. One of the more amazing things I heard talking to these developers is that

their research board said, oh, this tool can't lie. Even though deception is a common strategy in diplomacy, there was a research ethic issue when they were testing this with real people. They said, make sure it doesn't lie. Now, if this was just an LLM, we'd be like, I can't do that. I don't know. It just does what it does. It's hard to control. In a modular architecture, it was simple because they have a human-written planning engine that evaluates different moves. They just program that engine to, when it's evaluating moves, not to evaluate moves that involve deception. 100% will never lie. So when you move towards these modular architectures, part neural, part symbolic, a lot of these transparency issues, these control issues, these obfuscation issues, a lot of those go away. The bad news is these are hard to build.

You have to custom build them for particular problems. They're hard to get right. And if you're trying to build a company that's going to be worth $2 trillion, you want the answer to be, no, no, no, no. Just scale, you know, Fable 6 or GPT 7 and spend $500 billion on it. And that'll be the solution because there's a huge moat around that. You can, and your mind, you'll be able to build many different tools off the same model. There's a lot of economic incentives to wanting a massive LLM powering all types of agents to be the right answer. But if we put aside the economic concerns of the investors of those companies, it's not the safest, most effective way to build these type of tools, at least in my technical opinion.

We'll be right back. And for even more markets content, sign up for our newsletter at profgmarkets.com. He looked good. Then I thought, oh, now I'm going to show you someone who has an interest in terms of when will you see him? What do you hear from him? But he didn't have anything to lose. So after a couple of weeks I had my first five... I made a thousand euros over and then he would go trade for me. I always say that their brain is completely hacked. Maybe that I wanted to go with her story. How this goes? Listen to Herken Fraude, Voorkom Fraude. A podcast from the Netherlands Bank. code money power and love and who doesn't love all those things we are the internet's wall street and silicon valley big sisters tune in every monday for late night sister talk meets boardroom strategy check out tiger sisters on youtube spotify or wherever you get your podcasts welcome to the tiger fam as u.s americans voted in the midterm elections a lot was uncertain but one thing was clear this year many of the attention grabbing candidates identified or were identified as populists. Gina Hinojosa, a Democrat who ran a surprisingly competitive race for governor in Texas. Populism is about economic power. Sam Forstack, who ran for Congress as a

Dem in Montana. I'm running as a populist because populism is what the Democratic Party is supposed to be about. It's what democracy is supposed to be about. Josh Turek, a Democrat up for a congressional seat in Ohio. We want to once again have a common sense prairie populist that is fighting for the people. And Senate hopeful Dan Osborne, who ran as an independent in Nebraska and didn't use the word populist, but... My life sucks too. Costs are too high. The last 10 years in the U.S. have been defined by Trump-style populism. This year, candidates from across the political spectrum embraced the populist mantle.

On Today Explained from Vox, what has 10 years of populism gotten us? And are we ever going back? Today Explained drops every weekday. we're back with prof g markets looking into the future um when you sort of think about all of the risks involved i mean the lawsuits as an example the fact that we can't seem to nail down liability the fact that these chatbots continue to entertain our delusions and our psychoses also as you mentioned very expensive we haven't seen anthropics s1 yet but we know that last year they burned $8 billion on an operating basis. We know that OpenAI is burning even more, but the view from those companies is, we're going to reach such a level of scale that eventually it's going to work out and maybe we can raise prices. Though, of course, we're seeing a lot of pricing pressure too because of the open source models and because of China, et cetera. So taking into account all of that, is it your view that eventually we'll decide LLMs are not the answer and that we will revert to a more modular architecture? I think we're going to have lots of LLMs in our future.

I think it's probably bad news for OpenAI or Anthropic that they're not going to be the largest possible hyperscaled models. We're going to have plenty of models that are open-weight or cheap or even run on-device in a server rack in our offices that when combined with the right agents will do well enough. And we're going to have a lot more modular architectures for specific use cases because those are even cheaper, right? When you have these modular architectures, none of those components are massive. You actually get more of the intelligence around the interaction. This is why, for example, Stockfish, which can play chess, it's a chess engine that can beat Magnus Carlsen easily, right?

I can run it on a laptop, right? This is why the modular architecture that DeepMind built, they can learn from scratch how to play Minecraft well enough to actually find diamonds, which if you have kids, you know that's not easy to do in the game Minecraft. It fits on a single GPU chip. So I think we are going to see a lot more modular architectures because they're going to be better, be more controllable, and be cheaper. and we're going to see a lot of LLM-style tools similar to what we see now. They're also just going to be significantly cheaper because when you start to specialize, you don't necessarily need a 10 trillion parameter model like Fable 5 when what you're really doing is trying to control an Excel spreadsheet.

So there will be a lot of LLMs without modular architectures in the future, just with like smart agent frameworks around them. And I think a lot of modular architecture systems as well that will use LLMs among others. There's just going to be an explosion of diversity and what AI means in the near future. OpenAI and Anthropic, honestly, their bet was if we keep scaling LLMs, they will become so smart, they will achieve artificial general intelligence that there'll be no need to build any other AI technology. Just any piece of software you have can just ask this Oracle for whatever intelligence you need.

It's looking like that's not going to happen. That's not happening. The improvements of these have become more jagged. They don't like to talk about it this way, But if you really look at the last year, they really, OpenA and Anthropic, really confined their focus to three structured domains, cybersecurity, math, and computer programming because it was the three domains where they had a lot of structured data to continue to tune these models. Actually, the models are now doing worse on other things. They used to tout because when you overtrain for one thing, they get worse on other things. So it's become very jagged.

It's becoming more clear that this is not like it was with GPT-4 where the model gets bigger and it just gets better at everything. They're doing their best by looking at particular wins within those very structured domains to make it seem like the models are exploding in capability. I don't think that's actually true. They're not going to scale these models to some sort of artificial general intelligence. That was their bet. And I think that bet is not going to pay off. Why exactly do you believe that the models are not exploding in capabilities? I think some people who would push back would say, well, what about the fact that they solved this incredible mathematical problem that we hadn't figured out for many years and they won that prize they might point to like the video generation capabilities for example the will smith eating spaghetti is kind of the perfect example we look back to just a few years ago video if you prompted one of these video generators to create a video of will smith eating spaghetti he looked insane it looked kind of ridiculous and now when we look at it today it looks completely photorealistic um it's almost perfect so i think a lot of people would point it out and say no it

is exploding capabilities, why wouldn't that continue exponentially? Why wouldn't we reach superintelligence? What would be your response to that pushback? They have the wrong mental model for how AI capabilities actually increase. So I think the mental model that a lot of people have and the frontier labs are trying to push is that AI capability is like a water level that rises. This is a visual that came from Max Tegmark's book, Life 3.0, where you imagine that different types of challenges are like mountains of different heights in a landscape, and AI capability is a water level. And as that water level rises, it covers more of those mountain peaks. So now everything at this difficulty has been solved.

And as the water level rises, now everything at this difficulty has been solved. That's not actually how AI capabilities based on LLM are actually advancing. It's better to imagine it like tributaries going into a river. And as you explore certain tributaries, some of them, if you put enough resources into it, turn out to be very navigable. Like, oh, we were able to get deep down this particular tributary. But the key is successfully exploring one of those doesn't mean that you automatically are going to make progress on others. So this is what's known as jagged capability frontier, right? So you put a lot of resources into a lot of problems.

Some with enough resources turn out to be a good fit for this technology. Many others do not. That's the better way to explain this, is that there's a huge amount of exploration of these tributaries and some of them are proving navigable. Yeah, image generation, yes. Video generation, yes. But over in the non-video LLM realm, That's a slightly different type of generative model. But in the pure large language model realm, it's highly structured domains where you have a huge amount of information, including prompts with right answers. So you get this in computer programming, mathematics, and cybersecurity.

And where the underlying language that's being expressed is highly structured, like mathematics or programming languages, these are tributaries where we can explore really well. But there's other tributaries that we failed on, right? I mean, we were told at the beginning of 2025 that this would be the year in which the type of agents that computer programmers are using would now become widely adopted by all knowledge workers. It didn't happen. It turned out that tributary is much harder to explore than computer programming. They might brag that like, hey, we took this millennium problem that actually mathematical researchers had figured out the right approach, had solved a special case.

And then we took that approach and threw $22 million worth of compute at it to generalize it through brute force techniques and say, hey, look, we're really smart now, but why, if it was generally as smart as someone who could solve a millennium problem, why not solve problems that would be incredibly lucrative to businesses? Why not use that to make billions of dollars? Like, it's a pretty esoteric realm. I work in discrete mathematics. I know a lot of the problem spaces that has been solved recently by these LLMs. I appreciate it. I'll tell you who doesn't is like everyone else in the world. You can work on extremal combinatorical theorems, but is that where the money is? No, they're there because that's where you have a tributary you can explore. So the jagged frontier means most of the tributaries you explore might not turn up something. Some might if you put a huge amount of resources into it, but it's not very cost effective. Some will turn out to be really well suited. I really think if you look at the last year versus the two years before it, the two years before it, there was lots Lots of general applications they were talking about.

This last year has really focused on mathematics, cybersecurity, and programming and breakthroughs in those fields as a way to try to give the general sense that the water level in general is rising. And I don't think that's the right mental model. Let's look. it like enterprise usage as a tributary? Because it sounds like your view is, you know,

have enough runway to build these specialized tools? Or is it going to be, you know, a Clayton Christensen nightmare scenario for them where you have a thousand competitors from below working with specially tuned, low-cost models, and you just have a big environment of specialized tools that people pick and choose from, sort of like the software industry became in the 90s and early 2000s? What do you think the answer is? I hope it's that second answer. I mean, I often talk about what I think would be best in terms of AI progress is this view that I call distributed AGI, right?

So if AGI is a state in which AI can do most things that humans can do at a human level or above, to me, the best way to get there is to have a thousand different tools, each of which does one thing as well as a human or better. that's going to be much more economic diversity because these tools are going to be produced within the industries often that they serve as opposed to trying to consolidate this all in one or two AI companies. It's much safer because specialized tools, as we talked about, often with some modularity to their architecture, are very controllable. We don't worry right now about specialized tools going rogue for the most part when you're just centered on doing one thing.

And I think they'll be more effective, right? If you have people in your industry working as a labor of love that produce like the perfect AI tool for what you do, it's just going to work better than some sort of like thrown together software wrapper on top of a major hyperscaled anthropic. a limb. So this distributed AGI vision is one in which all of our lives is much better. Our economy as a whole is going to be a much bigger pie. Those rewards will be much more spread out among the economy and not consolidated, and we'll have many fewer safety concerns. I just think it's a much better vision than what OpenAI or Anthropic want, which is one ring to rule them all. We have one massive model, which is like in the Isaac Asimov story, a brain that we put in a box that powers the whole society.

That's a dangerous vision. That's an economically undesirable vision. That's a vision that hopefully is not going to win out. We'll start to wrap up now with regulation. This is becoming the ultimate question in Washington. How do we regulate AI? What are your top line views on what we should be doing in government? It's two parts, right? So I say part one, we need to better understand what's actually going on. This is why I wrote in my New York Times op-ed about this, that we probably need much more testimony and congressional investigation into the existing AI firms, especially open AI and Anthropic.

We need to go in there and find out what specific tools are causing problems, what are their safety protocols around them, and more importantly, what are their motivations? This is a whole other topic, but I do think the impact of utopian futurist ideologies on OpenAI and Anthropik and many of their critics, all of whom come from these same circles. It's just an inner scene squabble. It's a family fight about, you know, OpenAI and Anthropik think they're ready to create the good God and these other EA folks in the safety community are like, you're not ready yet, but they all ultimately believe the same thing.

We need to make that all transparent so we know what we're talking about. But from a legislative perspective, probably what's going to be important is strict liability. Okay, we need to update liability rules so that if you built this tool and this was a predictable consequence of this type of tool, you're liable for it. Ultimately, that is probably going to be the best tool we have to prevent the more dangerous behavior. Do you have a view on data center regulation? Because the increasingly popular position is that we should put moratoriums or even ban data centers because of how much people hate these things.

What is your view on data center regulation? Well, I think the market's going to help sort that out pretty soon as well. There is a lot of bubble activity going on in data centers where there's way more money now actually invested in data centers than we actually have, you know, chips and cooling and electricity to ever actually fulfill. So typically bubbles have a way of working themselves out when something pops. And I think that's definitely happening, definitely gonna happen around data centers. Also, popular governance is working well here. Local government pushback against local elected officials.

We don't want this. I think that can be very powerful. It's a hard situation in Washington right now. I mean, I talk to a lot of congressional staffers and they'll tell me, it's like, it's tough. Like an AI company can come in and be like, hey, we really want you to X, Y, and Z. By the way, we're gonna build a $20 million data center project in your district, right? Right. So it's also these data centers are being used as a political lobbying tool as well. But I think that's going to shake out when this bubble, the bubble of, hey, if we just put our money in data centers, we're going to get a big return.

That is not a sustainable investment scheme right now. When that pops, I think that's naturally going to govern the excitement around continuing to try to build too many of these. I've also pushed for we need more actual investigation, whether that's congressional testimony. I've also suggested we should just hand subpoenas to OpenAI and Anthropic, especially after all of the extinction debate where they just throw out there that there's a 10% chance that we're all going to be dead within the next decade. My view is, okay, really? Prove it. Show us. How did you reach that number? Give us the evidence. Some of the pushback that I've received is we don't have anyone competent enough in our government to ask the right questions and to do a proper investigation to get us anywhere. What is your view of that position, that even if we were to bring them in, even if we were to haul Sam Altman and Dario Amodei and Jacob Cox's arses to Washington, that we wouldn't really get anywhere? You know, I don't buy that. We've seen this before. Like, what happened with the Watergate hearings, for example? They pulled together a giant team of lawyers to aid the Senate committees to actually, for the hearings, right? You could bring in a huge

amount of experts, and they do. I mean, I talk to staffers all the time. I brief on the Hill. A lot of people I know do coming out of my, because I'm in D.C. You bring together a bunch of experts that can sort of create the testimony, to create the questions, to help vet them. We've done this before on all sorts of bigger issues. I think you're absolutely right about hauling people in and getting answers, especially around the extinction. I just think this is really important. I'll just say it real quickly, but it's something I've been writing about for a long time. And now a lot more reporters are also covering this.

I do not think neither Congress nor the American people understand the ideology. players involved here, that you have this ideological community that has this belief that ultimately superintelligent AI will deliver us a transhumanist utopia, but if we do it wrong, it'll kill us all. This community has been around well before LLMs were here. And I think what people don't understand is that this community is slightly bifurcated. So OpenAI and Anthropic come out of this community, but they believe that they can deliver the utopia, that we're ready to do it. The people yelling at them, the Jacob Coxsons of the world, also come from this community.

They just think we're not ready to do it yet. And so the entire debate right now is all being shaped by this singular ideological community. And now that community is investing huge amounts of money. They call them fellowships, but there's a large number of congressional staffers who have been placed on those staffs out of these communities to make sure that their views will be seen there. They're now giving out giant grants to journalists. And so I don't think we understand the pervasiveness of this particular transhumanist utopia ideal. We think the AI safety people is a different crowd that doesn't like the AI labs. They're all the same people. They just are squabbling among each other about whether we're ready yet to generate the AI gods. So I think we need to make that much more clear as well. We need to understand the influence of these communities, not just on the debate, but on the particular experiments that these companies are running. We need to understand, you know, how it changes their thought, the biases that introduces. All of this is starting to come out, but it's something that we really need to emphasize because we cannot clearly debate this issue if we do not understand where the various

involved parties are coming from ideologically. Why is it so important for us to understand that this is all kind of one ideological community? You call it this sort of transhumanist utopian community i mean the way i kind of see it when i look at all of these people debating and i agree they're all kind of from the same school of thought it's a lot of people who are really into science fiction and they read a lot of science fiction books and it kind of framed their whole worldview um and you can tell me if that's the wrong characterization or not but why is it so important for us, in your view, to understand that? What would it change about our current trajectory if we were all clear on that? I think there'd be three things to what happened. One, it would change the way we process the rhetoric coming from these people. We spent the last two years.

incredibly stressed out as a society because of rhetoric that is incredibly common in those circles. It started with every job is going to go away, and then it shifted over to we're all going to be extinct, and then we have particular percentages for extinction. And a lot of the media took that as these are the people who know the technology best, so if they're worried, we should be worried too. But if you are familiar with these ideologies, you'll say they've been saying this for 15 years. That gives you an asterisk to how you actually are going to think about it. Is it your view that the ideology is delusional or misguided, that they've been brainwashed by some ideology that actually isn't true? I mean, I don't know how to say true or not true, but I think it's a very questionable ideology. I think not everyone pushes it to the extreme, but when you push this ideology to the extreme, it's very anti-humanist. So it doesn't value the individual human life in the same way that most moral systems we're comfortable with you. It's very much an expectation-based ethic in which the means justify the ends.

That's why you'll have people like Eliezer Yachowski talking about it wouldn't be that bad to have a nuclear war that kills billions and billions of people if it helped increase the probability that in the distant future, we successfully upload digital consciousness to the stars. So there's a sort of anti-humanist expected value. We're valuing the 10 trillion lives in the future higher than the 8 billion lives that exist right now. Every movement in the recent history of the last century that has had a sort of ends justifies the means ethic seems to have ended up in pretty terrible ways. We don't look back at them and say, oh, yeah, that worked out okay.

This is both on the far left and far right. Movements have fallen in. So I think there's a sort of worrisome aspect to the actual ethic. I think it's also important to talk about that a lot of the things they're talking about now, they've been talking about for a long time, and it comes out of their circle. Like recursive self-improvement is not a computer science concept. It's a concept that comes out of futurist circles from the 80s. It was a way to talk about super intelligent AI if you didn't know anything about technology. You could say, well, it's not, I don't have to justify why this will work.

It'll just fix itself up. It's just an idea that just sort of emerged in these non-technical circles. So I do think it matters to know that's where they're coming from. And one of the things that might lead you to do, which I've been pushing for, is like, okay, well, here's an easy sanity check. if someone in AI circles who's connected to those ideologies say something that worries you, also check what people in the AI circles not connected to that ideology are saying? Are they saying the same thing? If not, that's a clue that this might be more ideological than real. And we've been seeing that with the extinction debate recently, right?

That we have large AI companies with hyperscaled LLMs who obviously know the technology very well, but aren't connected to futurism, basically perplexed and aghast at what they were hearing from Anthropic and OpenAI. That's why you have Mark Zuckerberg from Meta coming out and saying, hey guys, almost sarcastically in a tweet, Maybe if you stop thinking about recursive self-improvement and focused on building useful products, this wouldn't be an issue. This is basically a non-ideological—I mean, I have my issues with Mark Zuckerberg, but someone— Yeah, but he's looked pretty good in the whole AI debate so far.

Well, Jensen Wang's been the same way, and he has his own biases. He doesn't want export bans because he wants to sell chips. But you can see the frustration over the last year in his interviews where he'll say things like, I read sci-fi as a kid, too, but enough with the sci-fi. What that's really telling you is that the part of the AI industry who knows this technology that doesn't come from those circles is seeing this very differently than the part of the industry and the critics that all come from those same circles. So understanding the ideology, like I get accused sometimes of saying, well, you're just accusing them of being weird.

That doesn't mean they're wrong. And I don't care if they're weird. We're all weird. Jensen Wong wears a motorcycle racing jacket for some reason. We're all weird. Computer scientists are weird. It's not about weirdness. It's about specifically how does this ideology affect how you talk about your technology and the experiments that you run. And I do think those hacking experiments OpenAI run, they did them hastily because they felt like they were racing to superintelligence. And I think Anthropic's obsession with recursive self-improvement, which is going to create models and tools that are more obfuscated and more unpredictable, is not because they're trying to build better products, but because they're racing to create superintelligence.

So these ideologies affect how they talk and how they act. And so we have to understand it. Do you think that it is affecting the actual trajectory of America? And by the way, you say, you know, they're weird or that you're not necessarily accusing him of being weird. But I think that that actually might be the right word, or maybe kind of the word would be crazy delusional, because I think, to your point, what we have is like a pretty extreme, fantastical perspective on the world that is steering the direction in which supposedly the most important technology of our time are being built. And so maybe it is like...

you guys are all a little bit delusional. You guys are all a little bit, you're too into your sci-fi books and it's messing with your understanding of reality. But I guess the question is, do you believe this ideology that it is actually affecting not just the conversation, but the actual reality of the human and societal experience today? I think it really did speed up The rate at which we grew LLMs, the amount of money we're willing to invest in them, I think OpenA and Anthropic, racing each other, motivated in part by ideological goals, probably was moving for this technology faster than what would have happened if it had been discovered in other non-ideological circles where it might have progressed in more of a careful way of like the Google way, the Amazon way of like, let's try to build a product.

now let's build another. So for better or for worse, I think it has changed the speed at which the technology advanced. I think it led to investment in these sort of big showy things like solving math problems that really don't make sense because there's not a product there unless you're really trying to sort of race forward and demonstrate the intelligence of your model. So I do think it has changed things. The biggest impact I think it's had, though, is it's been a weapon of mass anxiety. It has been the singular cause of a massive amount of anxiety worldwide, but especially in America. And I say especially in America because the AI industry and other countries aren't connected to this futurism. In Europe, they don't think about it the same way.

In the CCP, which they do terrible things with AI over there. I don't want to defend the CCP. But they don't talk about existential threats or every job is going to be automated because that ideology doesn't exist. They wouldn't be allowed to exist in the CPT. It's too cult-like or religious. They wouldn't allow it to exist. So I think the biggest impact of the leading companies happening to be connected to this ideology is that it's been a weapon of mass anxiety. We undervalue how big of a harm that's caused. A lot of my work on this topic has come out of my frustration that everyone I know is now having to live under a blanket of anxiety because of the beliefs of these people in the Bay Area. And so I've been trying to be out there and be like, let's try to reality check this a little bit.

This is, you should care about this, but you shouldn't be anxious about it. And it's so much unnecessary anxiety after we had to go through the COVID years of year after year of actual justify anxiety. It just feels like from a humanist perspective to be a real wrong. And this ideology, I think, is a major driver of it. If those companies weren't involved in the generative AI revolution, we would still have really advanced tools and no one would be stressed out about it. That's my counterfactual. Cal Newport is a professor of computer science at Georgetown University, where he is also a founding faculty member of the Center for Digital Ethics and the inaugural director of the Computer Science Ethics and Society Academic Program. Cal writes about technology and its impact on our ability to live and work deeply in an increasingly distracted world. He is the New York Times bestselling author of eight books, including most recently Slow Productivity, a world without email, digital minimalism, and deep work. He's also a longtime contributing writer for The New Yorker and the host of the Deep Questions podcast. Cal, this was fascinating.

Thank you so much. Thanks, Ed.