# AI Cannot Reliably Run a Company for Six Months | Richard Ngo

Markdown copy of https://realalpha.app/v/YJjXjKJhUnI/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

Richard Ngo argues AI may make the world increasingly strange without a near-term intelligence explosion, while models remain unreliable at long-horizon work.

- Show: MTS
- Hosts: [Theo Jaffee](https://realalpha.app/people/theo-jaffee/) & [Sophia Dew](https://realalpha.app/people/sophia-dew/)
- Guest: [Richard Ngo](https://realalpha.app/people/richard-ngo/) is an independent AI researcher; former OpenAI governance and DeepMind safety researcher.
- Published: Nov 4, 2026
- Length: 32 min

## The best minutes

- [12:37](https://realalpha.app/v/YJjXjKJhUnI/?t=757) [Running a company exposes AI’s gaps in reliability](https://realalpha.app/s/running-a-company-exposes-ais-gaps-in-reliability/) (4 min)
- [19:19](https://realalpha.app/v/YJjXjKJhUnI/?t=1159) [Alignment research should clarify what good behavior means](https://realalpha.app/s/alignment-research-should-clarify-what-good-behavior-means/) (3 min)
- [22:31](https://realalpha.app/v/YJjXjKJhUnI/?t=1351) [AI safety needs groups willing to challenge powerful labs](https://realalpha.app/s/ai-safety-needs-groups-willing-to-challenge-powerful-labs/) (5 min)

## Also worth watching

- [5:00](https://realalpha.app/v/YJjXjKJhUnI/?t=300) [Rapid self-improvement timelines outrun the evidence](https://realalpha.app/s/rapid-self-improvement-timelines-outrun-the-evidence/) (5 min)
- [1:39](https://realalpha.app/v/YJjXjKJhUnI/?t=99) [AI safety's deference to labs reflects fear of conflict](https://realalpha.app/s/ai-safetys-deference-to-labs-reflects-fear-of-conflict/) (3 min)
- [10:30](https://realalpha.app/v/YJjXjKJhUnI/?t=630) [AGI and superintelligence are distinct concepts](https://realalpha.app/s/agi-and-superintelligence-are-distinct-concepts/) (2 min)

Skip the sponsor reads and the closing discussion of rationalists, elite social pressure and the 2024 election.

## Full analysis: Richard Ngo says AI may struggle to run a company for six months

- Ngo says models could struggle to run a profitable company without legal trouble for six months.
- That is an illustration of a difficult long-horizon task, not a measured benchmark.
- Teams should separate demonstrated capability from reliable execution and test costly failure modes.

Its 6 sections, what it means for you and what to watch are on [the page](https://realalpha.app/v/YJjXjKJhUnI/#analysis), free with an account:

- [14:00](https://realalpha.app/v/YJjXjKJhUnI/?t=840) Ngo uses a company run to illustrate a hard task
- [14:44](https://realalpha.app/v/YJjXjKJhUnI/?t=884) Elicitation can hide capabilities models already possess
- [8:00](https://realalpha.app/v/YJjXjKJhUnI/?t=480) Ngo rejects certainty about very fast self-improvement
- [11:09](https://realalpha.app/v/YJjXjKJhUnI/?t=669) AGI and superintelligence need separate definitions
- [20:17](https://realalpha.app/v/YJjXjKJhUnI/?t=1217) Alignment research needs clearer behavioral arguments
- [22:52](https://realalpha.app/v/YJjXjKJhUnI/?t=1372) Safety groups must scrutinize their own power

## Summary

Richard Ngo is an independent AI researcher who previously worked on AGI safety at Google DeepMind and governance at OpenAI. He is writing a LessWrong sequence on what he sees as failures of the AI alignment movement.

Richard Ngo argues that rapid recursive self-improvement is not established by current evidence, though he expects AI to make the next decade unusually strange. He says models may excel at some tasks while remaining unreliable at long-horizon work such as running a company, and that alignment can affect which capabilities people elicit. Ngo says safety work needs sharper concepts and explicit arguments about desired behavior, not reliance on labels such as alignment. He also argues that AI safety depends on candid, trusted groups that can examine labs’ power and resist political and organizational groupthink.

### Minute by minute

- [0:00](https://realalpha.app/v/YJjXjKJhUnI/?t=0) **Company operations demand robustness.** Richard Ngo says running a company for six months while keeping it profitable and out of legal trouble could be difficult for AI models. A larger company has a broad attack surface: hackers, rivals or one mistake could trigger legal trouble that shuts it down.
- [1:01](https://realalpha.app/v/YJjXjKJhUnI/?t=61) **Lab advocacy must confront power.** Ngo says effective altruism had historically been conciliatory toward AI labs because people feared challenging prestigious, powerful companies. He sees speaking against lab race dynamics as a positive step, while warning that good-versus-bad framing can obscure the power dynamics.
- [4:35](https://realalpha.app/v/YJjXjKJhUnI/?t=275) **Forecasts need more than fast takeoff.** Ngo says many people in the field are scared by the pace of progress and may still underestimate it, but struggle to imagine AI companies growing more powerful without recursive self-improvement. The host says claims that AI R&D will soon be fully automated and trigger an intelligence explosion go beyond the evidence; Ngo agrees.
- [7:56](https://realalpha.app/v/YJjXjKJhUnI/?t=476) **A strange future need not mean takeoff.** Ngo says the evidence does not justify expecting rapid, extreme self-improvement in the next few years. But he also rejects a normal future: he expects AI to make the next decade much stranger, even if no sudden jump into superintelligence occurs.
- [10:30](https://realalpha.app/v/YJjXjKJhUnI/?t=630) **AGI and superintelligence need distinction.** Ngo says people once used AGI as a broad term for very sophisticated AI and often conflated it with superintelligence. He argues the terms describe different things, and says superintelligence itself needs to be unpacked into more precise concepts.
- [12:37](https://realalpha.app/v/YJjXjKJhUnI/?t=757) **Capabilities will remain jagged.** Ngo says models are already close to superhuman at some kinds of math and may become superhuman at novel writing reasonably soon. He sees long-term company operations as harder because they require robustness against hackers, rivals and legal risks; reliably running a profitable company for six months could be difficult.
- [14:40](https://realalpha.app/v/YJjXjKJhUnI/?t=880) **Alignment can shape capability access.** Ngo says models may have capabilities that people do not know how to elicit. He points to chain-of-thought prompting, which revealed overlooked abilities in early language models after one to two years, and says models may show different capabilities depending on whether they trust the person or organization prompting them.
- [17:21](https://realalpha.app/v/YJjXjKJhUnI/?t=1041) **Alignment research needs clearer concepts.** Ngo says labels such as alignment are too imprecise to show whether research advances safety or capabilities. He favors clarifying concepts and making explicit arguments for what good AI behavior should look like, citing disagreement over how to interpret Anthropic’s alignment-faking paper and the rationale for Claude’s constitution.
- [22:11](https://realalpha.app/v/YJjXjKJhUnI/?t=1331) **Safety work depends on trusted groups.** Ngo says small groups need trust and clear communication to coordinate and grow their influence. He argues that early OpenAI had more serious AI discussions than mainstream ML or most of DeepMind, and says this helped OpenAI out-compute DeepMind before Anthropic out-computed OpenAI.
- [25:00](https://realalpha.app/v/YJjXjKJhUnI/?t=1500) **Ngo says Anthropic should examine its influence.** The host suggests Anthropic fits Ngo’s model of a coordinated group. Ngo counters that groups can cohere around dogmas, including the belief that others would build the technology anyway; he says Anthropic should assess its own influence, including fears he attributes to some Chinese researchers.
- [27:16](https://realalpha.app/v/YJjXjKJhUnI/?t=1636) **Ngo says elite pressure narrows debate.** Asked about his 2024 support for Trump, Ngo says rationalists’ ties to cultural elites and social pressure can make dissent costly. He argues that AI urgency pushed some to avoid political disagreement for fear of losing influence, a strategy he says impaired clear thinking about AI plans.

## Transcript

MTS, hosted by Theo Jaffee and Sophia Dew, with Richard Ngo.

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[0:00](https://realalpha.app/v/YJjXjKJhUnI/?t=0) the sort of thing that feels more difficult is running a company for long periods of time and when you're running a company especially a larger company you just have this enormous attack surface where you know one mistake uh can be exploited by a bunch of adversaries like hackers your uh your rivals you can mess up and then be in trouble with the legal system in a way that shuts down the entire company uh so all of those things prioritize a certain level of robustness that the models don't currently have. And my sense is that it could be pretty difficult to get the models actually reliable on a task like run a company for six months and make sure it's profitable and doesn't end up in legal trouble.

[0:43](https://realalpha.app/v/YJjXjKJhUnI/?t=43) And we are back. We are live with Richard Ngo, who is an independent AI researcher who worked on AGI safety previously at Google DeepMind and on governance at OpenAI. before leaving to become independent. Richard, welcome to MTS. Thanks for having me. Absolutely. So why don't we talk about, start with, there seem to be a lot of people now in the mainstream EA world recommending resigning from the labs, making things more difficult for the labs. Like Robert Wiblin, for example, from the 80,000 Hours podcast just posted this long video today, actually, about how people have been able to get the labs to change behavior basically through activism.

[1:30](https://realalpha.app/v/YJjXjKJhUnI/?t=90) And you've been talking about this for some time. You've been recommending people resign from labs. Do you think that this is a positive change from the mainstream of EA? That's right. I think that EA historically has been very conciliatory towards the labs

[1:50](https://realalpha.app/v/YJjXjKJhUnI/?t=110) **Richard Ngo:** And not, I think, for a particularly good reason, just because people were somewhat conflict-averse and scared of challenging, prestigious, powerful companies.

[2:02](https://realalpha.app/v/YJjXjKJhUnI/?t=122) So at the very least, being willing to do that is a pretty good sign. I do think we're still not quite at the point where people are willing to actually analyze the power dynamics involved in labs and try to be honest about what's actually driving them. But at the very least, being willing to speak out against some of the racing that they're doing seems pretty seems like a pretty good step. Right. And similarly, Scott Alexander just wrote a post called was called against. I'm forgetting the acronym. Sura. Sura. Yes. Yes. Against Sura, which is the strategy of stop unilaterally and raise awareness.

[2:44](https://realalpha.app/v/YJjXjKJhUnI/?t=164) A lot of people were saying this is basically a, you know, self-deceptive defense of Anthropic, where Anthropic is, you know, against Sura itself. It is not stopping unilaterally and raising awareness. It is continuing to push the frontier to some extent. Do you think that this is a good thing for Anthropic to be doing? Yeah. Yeah. So interestingly, a lot of the time when people discuss this stuff, they slip into this very almost basic moral reasoning of like good guys versus bad guys. And sometimes the good guys are, you know, OpenAI cast themselves as the good guys against DeepMind early on.

[3:24](https://realalpha.app/v/YJjXjKJhUnI/?t=204) And then Anthropic cast themselves as the good guys versus OpenAI and possibly versus China as well. my sense is basically that i you know i i think there are a bunch of people who are trying quite hard to be serious about the stuff in uh many of the labs and somehow there's just this very yeah very basic in some ways almost childish reasoning about um you know like trying to beat your opponents trying to get there first i'm not saying it's it's fully wrong but But... It's very interesting that it's the kind of trap that we see throughout history, and I think people have reasoned their way into thinking that this case is different, that in this case, you actually need to race ahead and beat your opponents.

[4:15](https://realalpha.app/v/YJjXjKJhUnI/?t=255) This time, everything's up for grabs because AGI is such a powerful technology. uh so that's my rough sense of what's going on that there's a kind of um that there's some kind of like they hit the point of um the standard tribal reasoning that people get into and then they sort of stop and it's it's hard to get beyond that point right um you wrote this uh or you're currently writing this sequence of posts on less wrong uh as a retrospective against the failures of the AI alignment movement over the last decade or so. Do you think that the community has responded at all to this? Do you think they have, have you seen any signs of change towards what you would want them to be doing?

[5:04](https://realalpha.app/v/YJjXjKJhUnI/?t=304) Not a huge amount. My sense is that there are a bunch of people who are reasonably thoughtful and open-minded, but tend to be in a pretty scared and panicky mindset because of the rate of AI progress. And so it's very difficult because, you know, almost everyone throughout the last, over the last decade or two has underestimated the rate of AI progress. And really a lot of the, you know, a lot of the forecasts, a lot of the capabilities we see right now are pretty sci-fi compared with what people expected five or ten years ago and so it does feel very hard to rule out the possibility that people are still dramatically underestimating the rate of progress at the same time i i sort of get a sense that a lot of people can only consider that possibility or they sort of they don't have a picture of a world in which uh things go more slowly and um you you get you know the something like the standard situation the current situation uh moving forward for the next decade or two decades with the ai companies becoming increasingly powerful and ai becoming increasingly pervasive but without some you know hitting the wall of recursive self-improvement so that's a lot of the scenario that i'm trying to think about like

[6:30](https://realalpha.app/v/YJjXjKJhUnI/?t=390) what if you know in a decade or two decades um ai is by far the biggest thing in the world but But at the same time, there hasn't been a single point where the lines went vertical, basically. Right. Yeah, I mean, I've also been thinking about this a lot. It seems to me that there's a remarkable amount of jumping to conclusions around short timelines and around very high PDU. That seems to me to be unjustified. It seems like people are taking the idea that AI labs are increasingly applying models to automating parts of AI R&D to mean that in very short order, the entirety of AI R&D will be fully automated.

[7:13](https://realalpha.app/v/YJjXjKJhUnI/?t=433) It seems like they're taking that to mean that there will be a rapid self-sustaining intelligence explosion with a high degree of certainty. it seems like they are taking all this to mean that with you know pretty high confidence ai timelines to you know for real for real super intelligence as opposed to merely more powerful versions of the jagged and imperfect ais that we have today are very very short and like it it just seems to me that with the data that we have about recursive self-improvement and how this carries over to intelligence explosion dynamics, like this is just not justified under the evidence.

[8:00](https://realalpha.app/v/YJjXjKJhUnI/?t=480) **Richard Ngo:** I think so. So there's I agree that people are jumping further than the evidence warrants towards, you know, strongly expecting a very fast RSI of next couple of years.

[8:17](https://realalpha.app/v/YJjXjKJhUnI/?t=497) um i do have a sense that uh it's easy to polarize on this issue and say well you know like some people are saying that um there's going to be um a foom happening pretty soon um they're unjustified therefore you know things are going to be much more normal than they expect i i don't really see the world looking normal over the next decade or so um and and so the the default sense i i have for what's going to happen is that uh we're not going to get uh like yeah things going vertical if you will we're not going to get like sudden and extreme um self-improvement that goes far beyond uh that goes very quickly far beyond into the super intelligent regime but at the same time things are going to go much uh things are going to look much weirder than almost anyone expects uh and yeah the exact shape of what it looks like for the world to be very weird is up for grabs and i think nobody's done an amazing job at describing that but right there's some i almost want to say that there's some point at which the anthropic and other folks who expect uh very rapid improvement are going to be directionally correct and sort of like uh maybe you could say morally correct if you will like to to expect that um things are going to get really crazy

[9:40](https://realalpha.app/v/YJjXjKJhUnI/?t=580) It's just that there's a lot of room for things to get crazy, as there already has been a lot of room. The AI situation is very weird and strange compared with what anyone expected, compared with what most people expected a decade or two ago. Yeah. I want to shout out our sponsor, Lovable. You know the app you've been meaning to build, the internal tool, side project, or product you'd otherwise lose a weekend to? With Lovable, you get software you can ship today. That includes an editable codebase you can inspect and change, plus two-way GitHub Sync. Lovable also handles the backend and infrastructure, including managed Postgres, auth, storage, hosting, payments, and more, so you don't have to wire it all up yourself.

[10:21](https://realalpha.app/v/YJjXjKJhUnI/?t=621) And through Lovable's MCP server, you can create and deploy your project directly from the agents you already use. Turn ideas into software people love with Lovable. Lovable.dev. Now, on to the episode. So you wrote a, you know, short form post about this on LessWrong. Daniel Cocotagelo responded and you had like a back and forth thread. And Daniel basically said, well, like your claim doesn't really mean anything because things in the future will be so weird such that to the average person or to, I guess, any forecaster, It will be indistinguishable from, you know, the effects of true superintelligence, even if the machines that we're dealing with are not exactly true superintelligence.

[11:09](https://realalpha.app/v/YJjXjKJhUnI/?t=669) Yeah, my sense is that. So one way I put it is that people used to use the word, the term AGI as a catchall for, you know, incredibly sophisticated AI. And they didn't really distinguish much between AGI and superintelligence, partly due to arguments about recursive self-improvement. Like once, like probably there's not a huge gap between, say, a human level AI and a superintelligent AI was the argument.

[11:37](https://realalpha.app/v/YJjXjKJhUnI/?t=697) **Richard Ngo:** And I think over the last five years, let's say, it's become apparent that AGI is very different from superintelligence.

[11:45](https://realalpha.app/v/YJjXjKJhUnI/?t=705) And that we, yeah, like people who conflated those two terms have mostly ended up being confused. I think they'd say, well, you know, when we said AGI, we kind of meant superintelligence. But like, you need to get the terminology, like precision with this kind of stuff is important. So my sense is that the term AGI has now been like, we have much more clarity on what that means and what that looks like. And now the term superintelligence has replaced that as the thing that is sort of this, you know, catch all term for incredibly powerful AI. We don't know exactly what we mean by superintelligence, but we're sort of bundling together a whole bunch of different concepts.

[12:26](https://realalpha.app/v/YJjXjKJhUnI/?t=746) uh and my sense is that over the next few years we're going to have to unbundle superintelligence as a term and we're going to see like yeah like even the concept of superintelligence is going to need to get quite spiky is another way of putting it right so you think that like when you talk about how you know jaggedness could persist over a relatively long period of time are you imagining something like a model that is like wildly extremely superhuman at math and theoretical physics research and such that still can't really write a novel or something yeah i think novels are probably yeah let's see so models are already pretty close to superhuman on math um in some ways they are superhuman in other ways they're still falling short uh writing i expect them to be superhuman novel writing uh reasonably soon as well uh just because it's the kind of task that doesn't take

[13:28](https://realalpha.app/v/YJjXjKJhUnI/?t=808) **Richard Ngo:** that much feedback from the world the sort of thing that feels more difficult is running a company for long periods of time uh and when you're running a company especially a larger company

[13:38](https://realalpha.app/v/YJjXjKJhUnI/?t=818) you just have this enormous attack surface where you know one mistake uh can be exploited by a bunch of adversaries like hackers your uh your rivals you can mess up and then be in trouble with the legal system in a way that shuts down the entire company. So all of those things prioritize a certain level of robustness that the models don't currently have. And my sense is that it could be pretty difficult to get the models actually reliable on a task like run a company for six months and make sure it's profitable and doesn't end up in legal trouble. Yeah, it does seem to me on the topic of people jumping to conclusions beyond where the evidence is warranted, One such area is the idea that, you know, every AI capability is just emergent.

[14:24](https://realalpha.app/v/YJjXjKJhUnI/?t=864) And, you know, if you automate AI research, if you have models working to overcome their own shortfalls, then the areas in which the models are jagged in a bad way will like just go away, basically. And this seems to me to be like plausible, but far from certain. Yeah. I also want to distinguish between capabilities that exist within the models and capabilities that we're actually able to access. So right now, I think there are probably a bunch of capabilities that the models have that we just don't know how to get out of them. You could call this an alignment problem or an elicitation problem.

[15:02](https://realalpha.app/v/YJjXjKJhUnI/?t=902) A historical example is just simple things like saying think step by step. were able to elicit capabilities from early language models that, you know, were overlooked for quite a long time, I think one to two years. And so, yeah, I feel like there's going to be some point at which capabilities and alignment as a concept become harder to distinguish because alignment is, in fact, the bottleneck on eliciting the capabilities. maybe for example models will only demonstrate certain capabilities when they trust the person who's trying to elicit it from them um and uh that might in fact already be true it's a little hard to say the models are pretty aware of uh you know the context in which they're being prompted who's asking them to do stuff and i wouldn't be surprised if um if models for example if claude for example, had a judgment of Anthropic overall that helped shape which capabilities it was displaying.

[16:07](https://realalpha.app/v/YJjXjKJhUnI/?t=967) And if Anthropic behaved in a more disreputable way or a more trustworthy way, then that might change the capabilities that Claude is willing to showcase. We'll continue monitoring right after this message from our sponsors. Scale your startup on Neon. Millions of developers and startups have already chosen Neon for their backend. Start on the free plan or get up to $100,000 in credits for your startup at neon.com slash MTS. Special thanks to our sponsor Kong, the AI connectivity platform. Connect APIs, LLMs, agents, and systems with serious security and governance. konghq.com. Finally prove what your AI investment is worth with Lairdyn. Head to lairdyn.com today to start measuring your AI's bottom line. MongoDB, trusted to run critical applications by 75% of the Fortune 100, the intelligent data platform for the era. Apply today for access to up to $15,000 in Atlas credits, dedicated technical support, and exclusive events and go-to-market opportunities.

[17:17](https://realalpha.app/v/YJjXjKJhUnI/?t=1037) MongoDB.com. Chargebee, billing and monetization infrastructure for the AI era. Chargebee.com. so you talk about alignment as a bottleneck on capabilities um this seems to be sort of different from how uh or rather you you talk about alignment as a bottleneck of capabilities almost as a positive thing here am i getting that right whereas elsewhere you've talked about you know the idea of uh prosaic alignment being a bottleneck on capabilities as like uh indication that the entire field of AI alignment has been pursuing the wrong things, right? Yeah. My sense is mostly that, again, we've reached a point where these words are too low resolution.

[18:07](https://realalpha.app/v/YJjXjKJhUnI/?t=1087) And so we just have these categories that don't quite make sense. And this is sort of the default you should expect from trying to forecast the future, that you're going to come up with these concepts and they're sort of not really going to be quite right. And over time, the gaps between the concepts and reality, the concepts are going to end up with a bunch of holes in them or the edge case is not going to be very well defined. And so like we have with AGI, right, like whether or not we have AGI already is in many ways a definitional debate. And the definition that people used in the past wasn't typically precise enough for us to actually have a clear resolution to that debate today.

[18:48](https://realalpha.app/v/YJjXjKJhUnI/?t=1128) So, yeah, I guess the main point I want to make is that we're going to need to think much more carefully about the kinds of research that are actually valuable and not. And just because something has historically been called alignment doesn't mean very much. That could mean that it's actually one of the things that advances capabilities the most. uh that that all feels very up for grabs especially given that the companies are using these terms in a way that um you know tries to sneak as much as possible under the banner of alignment right can you go into a little more detail about which uh areas of research or research directions you think are least productive and most productive yeah um so my sense is that most things that people are doing which rely on using uh the very frontier models are not gonna generalize that far and so we've had enough time with llms right uh so far that if if you're doing research that is going to generalize from current models to the ones that exist in a year or two then probably it will also generalize from current open weight models to the ones that exist in a couple of years.

[20:08](https://realalpha.app/v/YJjXjKJhUnI/?t=1208) **Richard Ngo:** So generally, I don't buy this idea that you need to be in the labs using frontier models in order to do good research.

[20:17](https://realalpha.app/v/YJjXjKJhUnI/?t=1217) The kinds of things that I am most excited about involve clarifying the sort of concepts that we're using. So basically, we have a range of concepts that were developed over a period of a few years, especially on LessWrong, that are now steering the decisions of very large companies and soon are going to be steering the decisions of major governments in pretty sensitive ways. and mostly my sense is that the concept is just not up to scratch and that when people say things like a good example here might be the alignment faking paper from anthropics so in that paper they had um you know claude was demonstrating some behavior that depending on how you uh define alignment could be very aligned or very misaligned it's just like people could uh the the term could just be used either way, basically. And so my sense is that the sort of research that feels really robustly good right now is the kind that tries to specify, for example, what good behavior would look like. Claude has a constitution that's led it to behave pretty well in a bunch of ways.

[21:34](https://realalpha.app/v/YJjXjKJhUnI/?t=1294) But what's pretty interesting is that I don't know if anyone has actually argued for the ideas in that constitution i don't know if anyone's sort of written down like here is how we think um an ai should behave and here's why uh claude behaving like this at scale is actually going to be good for the world or if they've done this it's sort of more like a pr type thing i don't know if anyone's actually given a um like coherent clear argument for that and that's that's the kind of thing that uh you know many people could could have done it and uh it's just the difficult kind of research to do because tangling with these sort of philosophical concepts is not incentivized very much by most people. Right. So what do you think the actual plan should be for the AI safety movement in general? Obviously, working on alignment research that would produce actual fundamental alignment of the models is a good one. But, you know, there's other components to this.

[22:35](https://realalpha.app/v/YJjXjKJhUnI/?t=1355) There's policy advocacy. There's lab advocacy. There's the decisions that the labs themselves should make. What do you think people concerned about making AI go well should actually do beyond just research? my sense is that uh the larger a group of people uh a lot of the stuff is going to be bottlenecked by groups of people who can trust each other and actually work together um and like share information clearly and and this is not the kind of thing that you see very much of in uh politics for example it's just like a lot of distrust and like people aren't really willing to think very much about uh they they often end up falling in behind the party line um and my sense is that the story of ai over the past few decades has been that if a small group of people who are thinking more clearly about um factual issues like uh you know the path towards building agi a small group of people can actually coordinate extremely well and then end up gaining a lot like growing a lot and gaining a lot of capabilities to influence the world but you really need that foundation of like can we uh like are people actually trying to communicate clearly with each other.

[24:00](https://realalpha.app/v/YJjXjKJhUnI/?t=1440) People at OpenAI in the early days were actually trying to have serious conversations about AI to a much greater extent than people in mainstream ML or even at DeepMind. Most of the people at DeepMind, I'd say. And that led to massively compounding benefits. OpenAI out-computed DeepMind and then Anthropic out-computed OpenAI. So my sense is that most people should be orienting to who's being honest and who's actually trying to navigate the power dynamics in a way that is much less scared of getting on powerful people's bad side than the field has been historically. and uh you know insofar as they can uh find people doing that then uh then you can start like following a plan but like before that you just don't have a coherent group of people who are going to actually be able to follow a plan yeah i mean like anthropic for example seems like kind of like what you are describing they seem to be good at uh coordinating with one another internally they seem to be very coherent they seem to be um very good at you know accessing and uh accessing power and increasing their own power it kind of seems like this is like this this this meets the criteria that you're describing and they have you know reached the point at which

[25:30](https://realalpha.app/v/YJjXjKJhUnI/?t=1530) they should start following a plan of sorts no the difficult thing is that the fastest way to get such a group to grow is by coordinating around uh sort of like sacred truths if you will like things you're not really allowed to question or um like things you're not allowed to look at so like historically something like christianity for instance had sacred concepts and uh dogmas actually i should say um my sense is the dogmas at anthropic are involve something like other people will do it anyway um or maybe you know like where the good guys and they're like and then like you have various narrative you have various like pieces of evidence that are marshalled to support where the good guys and like you know if it weren't for us other people would be doing anyway therefore you know it's it's fine to raise a head or it's fine to be less responsible and so on given that anthropic is not really able to accurately evaluate uh what other people would do or what other companies or countries would do then it's very hard for them to plan I think

[26:37](https://realalpha.app/v/YJjXjKJhUnI/?t=1597) **Richard Ngo:** just because they they aren't willing my sense is that Anthropic as an organization is not willing to look at its own influence on the world and for example the fact that I think many people

[26:49](https://realalpha.app/v/YJjXjKJhUnI/?t=1609) in China or many researchers are actually becoming pretty scared of Anthropic in particular and you know that's the kind of thing that you really need to be able to take into account when trying to make a plan um so yeah my hope for anthropic would be something like it's able to accurately evaluate this this idea of other people would do it anyway you know therefore um we we can get to do what we want and still be the good guys and somehow look past that right So one other question is that in 2024, you are like one of the very few people in the broader rationalist space to publicly back Donald Trump in general politics and specifically, you know, the binary decision on whether to vote for Democrats or Republicans is something that very smart people operating even very similar models of the world can differ on.

[27:45](https://realalpha.app/v/YJjXjKJhUnI/?t=1665) like they can come to uh one of these two choices or the other uh what do you think are the cruxes between you and the rest of or most of the rest of the rationalist movement on this in particular my sense is that rationalists in general are cultural elites um or like very closely affiliated with cultural elites and by cultural elites i mean here like you know harvard graduates or like top university graduates and then a sort of class of people that has gone under various names like professional managerial class is one term or class x is another term that's been used to describe the sort of current cultural ruling class of america and it's just difficult to go against your class interests if you're like it's very scary to face repercussions from the people around you and i think you sort of see this with the level of elite hysteria over brexit and over trump's original election where really there was some sense of people being whipped into a frenzy in a way that was hard to, like, wasn't really very grounded in specific things that they were worried about.

[29:03](https://realalpha.app/v/YJjXjKJhUnI/?t=1743) It was much more a sense of, like, the elite class is mobilizing against the working class, especially the white working class. And so, like, my basic sense is that, you know, most people who have this kind of intellectual background in the educator class are just going to be held in place by ties of, you know, ties of ideology combined with social pressure. And rationalists were good at breaking out of some of these, for example, by thinking about AI, even when that was seen as like weird, but other pressures are sort of scarier and harder to speak out against. And in some ways, like urgency about AI, I think, made rationalists less willing to think about the political situation. It's like, well, if you've got this big priority, which is AI, then you shouldn't rock the boat too much in other ways because that will lose influence. And unfortunately, it seems to have turned out that this was a bad strategy, that you really needed to be thinking clearly about all this stuff in order to make good plans about the future of AI. Right. Well, we're coming up on time, but it's been very interesting to have you on.

[30:21](https://realalpha.app/v/YJjXjKJhUnI/?t=1821) I think you're one of the clearest thinkers on a lot of these issues. There are a few people who are able to sort of escape the kind of groupthink bubble that surrounds many in AI safety. So much respect. I can't wait to read the rest of your sequence on alignment. Thanks so much for coming on MTS. I enjoyed the conversation. Thank you. A huge thanks to MTS sponsors. Blitzy, autonomous software development for enterprise code bases. Ship 5x faster. Blitzy.com. AdQuick, make your brand a billboard. Out-of-home advertising as easy to scale as digital. AdQuick.com. Arena, measuring AI performance in the real world.

[31:04](https://realalpha.app/v/YJjXjKJhUnI/?t=1864) Arena.ai. Support for the show comes from VCX, the public ticker for private tech. The U.S. stock market started history's greatest wave of wealth creation. From factory workers in Detroit to farmers in Omaha, anyone could own a piece of the great American companies. But today, our most innovative companies are staying private longer, which means everyday Americans are missing out. Until now. Introducing VCX, a public ticker for private tech. Visit GetVCX.com for more info. That's GetVCX.com. Carefully consider the investment material before investing, including objectives, risk, charges, and expenses.

[31:41](https://realalpha.app/v/YJjXjKJhUnI/?t=1901) This and other information can be found in the Innovations Fund Prospectus at GetVCX.com. This is a paid sponsorship.