Free with a sign-in.
The best 12 min of 32
AI Cannot Reliably Run a Company for Six Months | Richard Ngo
Richard Ngo argues AI may make the world increasingly strange without a near-term intelligence explosion, while models remain unreliable at long-horizon work.
MTS · Theo Jaffee & Sophia Dew
- Richard Ngo is an independent AI researcher; former OpenAI governance and DeepMind safety researcher.
The best minutes
- 12:37Running a company exposes AI’s gaps in reliability4 min
- 19:19Alignment research should clarify what good behavior means3 min
- 22:31AI safety needs groups willing to challenge powerful labs5 min
Also worth watching
- 5:00Rapid self-improvement timelines outrun the evidence5 min
- 1:39AI safety's deference to labs reflects fear of conflict3 min
- 10:30AGI and superintelligence are distinct concepts2 min
Skip the sponsor reads and the closing discussion of rationalists, elite social pressure and the 2024 election.
Full analysis · 6 min read
Real Alpha · Nov 7
Richard Ngo says AI may struggle to run a company for six months
- Ngo says models could struggle to run a profitable company without legal trouble for six months.
- That is an illustration of a difficult long-horizon task, not a measured benchmark.
- Teams should separate demonstrated capability from reliable execution and test costly failure modes.
Free with an account
Read the rest of the analysis
6 sections, the 7 other voices it tests Ngo against, and what it means for you. Every episode we cut gets one.
Check your inbox
We sent a link to your address. Click it to finish signing up. Not there in a minute? Look in spam.
or
Signed up before? The same buttons sign you in. The episode, its moments and the transcript stay free.
Ngo uses a company run to illustrate a hard task
Ngo uses a company run lasting six months as an example of difficult work. He says models could find it hard to do reliably.
He adds that the company would need to stay profitable and avoid legal trouble. One mistake, he says, could expose a larger company to hackers or rivals. It could also trigger legal trouble severe enough to shut the company down.
So this task combines a long horizon, adversarial exposure, and expensive mistakes. It differs from demonstrating one narrow skill. The interview supplies no failure rate, company size, or operating conditions.
Diogo Almeida, TypeSafe's co-founder and CEO, made a related engineering point on The TWIML AI Podcast in October 2026. He said Waymo decomposes systems because models remain unreliable, while demos can return work to humans.
Danila Shtan, Nebius's chief technology officer, spoke on Unsupervised Learning in November 2026. He said adoption depends on how reliably agents reach outcomes, not simply model size.
“run a company for six months and make sure it's profitable”
Diogo Almeida · The TWIML AI Podcast · Oct 6Waymo's reliability comes from decomposed engineering- DSDanila Shtan · Unsupervised Learning · Nov 2Agent reliability may matter more than model size
Our read
Treat the six-month company scenario as a difficult case Ngo raises, not a validated benchmark. It focuses attention on reliability, error costs, and exposure without giving teams a pass threshold.
Elicitation can hide capabilities models already possess
Ngo separates abilities a model may already have from abilities people know how to elicit. For example, he says asking a model to think step by step exposed capabilities people had missed in early language models.
He thinks those capabilities went unnoticed for one to two years. That is an elicitation gap, not proof that the model can sustain work reliably over a long period.
But Ngo also raises a more speculative possibility. He says models may notice who is prompting them and respond differently depending on trust.
He would not be surprised if Claude's judgment of Anthropic shaped which capabilities it shows. He speculates that a more or less trustworthy Anthropic could change what Claude is willing to demonstrate.
Our read
Elicitation and long-horizon reliability are separate problems. Ngo's suggestion about a model's judgment of its interlocutor remains a possibility he raises, not evidence that models already behave this way.
Ngo rejects certainty about very fast self-improvement
The host laid out a chain from models automating parts of AI R&D to all R&D, then rapid self-sustaining improvement. Ngo's agreement was narrower.
He said expectations of very fast recursive self-improvement in the next couple of years go beyond the evidence. He did not conclude that progress will look ordinary.
Ngo expects the coming decade to be much stranger than many people expect. He does not expect a sudden vertical leap straight into superintelligence, while leaving the shape of disruption open.
Herbie Bradley, an AI researcher and writer, said on MTS in November 2026 that evidence for imminent self-sustaining research is limited. He pointed to data dependencies as one possible bottleneck.
Matan Grinberg, Factory's co-founder and CEO, said on Sabrina Halper Show in September 2026 that recursive improvement could still fail to outperform human-assisted improvement.
Qasim Wani, Extrapolate's CEO, discussed the problem on MTS in November 2026. His account says RSI requires agents to handle data, experiments, evaluations, and infrastructure, not just model training.
- HBHerbie Bradley · MTS · Nov 7Fast takeoff remains a low-probability scenario
Matan Grinberg · Sabrina Halper Show · Sep 2Recursive self-improvement does not guarantee runaway growth- QWQasim Wani · MTS · Nov 6Recursive self-improvement means inventing a new lab
Our read
These comparisons identify separate steps that forecasts need to support. They do not establish that self-improvement will stall, only that partial AI research automation does not demonstrate a self-sustaining loop.
AGI and superintelligence need separate definitions
Ngo says people once used AGI as a catchall for extremely capable AI. He thinks expectations about recursive self-improvement helped blur AGI and superintelligence.
Now, he expects superintelligence itself to be divided into more precise concepts. Broad labels can conceal disagreements about what a system can do, especially when boards or policymakers must act on forecasts.
Take his example of models that are already superhuman in some ways at math, while still falling short in others. A model can excel on math tasks and remain unreliable at company operations.
The practical implication is to assess specific tasks and failure costs, rather than assume one broad label predicts performance everywhere.
Our read
Ngo's terminology point supports more specific evaluation, but he does not supply the definitions teams should adopt. Ask which capability is meant, under what conditions, and with what consequences for failure.
Alignment research needs clearer behavioral arguments
Ngo suspects much research that depends on frontier models will not generalize very far. He does not think researchers must work inside labs with frontier access to do useful work.
He is more interested in clarifying concepts that already steer company decisions and may soon steer governments. His example is Anthropic's alignment faking paper, where the same behavior could look aligned or misaligned depending on the definition.
Ngo says he does not know of a clear argument for the ideas in Claude's constitution. In particular, he wants reasons for why those behaviors would be good for the world at scale.
So teams can ask researchers to define desirable behavior and explain why it is desirable before calling a system aligned. That makes disagreements inspectable, even if the argument remains unsettled.
Ngo is an independent researcher and former OpenAI governance and DeepMind safety researcher. He is writing a LessWrong sequence critical of the alignment movement, so this is his diagnosis rather than an independent audit.
Our read
This is a case for explicit reasoning alongside empirical safety work. The episode does not show that Claude's constitution is wrong, only that Ngo sees its justification as unclear.
Safety groups must scrutinize their own power
Ngo says safety work depends on groups that trust each other, share information clearly, and can challenge powerful organizations. Without that foundation, he doubts a movement can follow a coherent plan.
He links serious early conversations at OpenAI with later compute gains over DeepMind, and says Anthropic later out-computed OpenAI. The interview offers no comparative measures for that causal account.
The host suggests Anthropic may already fit Ngo's description of a coherent group. Ngo pushes back, saying internal coordination can rest on assumptions people are unwilling to question, including that competitors would race anyway.
He says Anthropic needs to examine its own influence and its assumptions about other labs and countries. That means evaluating whether its choices affect others, not only whether its intentions are good.
Greg Brockman, OpenAI's co-founder and president, described personal ties and joint letters across rival labs on Odd Lots in September 2026. That is a narrower cooperation example, not evidence that trust lets groups confront lab power.
David Sacks separately criticized Anthropic on All-In in October 2026 for racing toward full RSI while calling for pacing. That partly overlaps with Ngo's criticism of racing logic, but it is a separate argument.
Greg Brockman · Odd Lots · Sep 14AI labs can build trust through small shared commitments
David Sacks · All-In · Oct 30Anthropic’s RSI push may split its safety allies
Our read
Ngo's organizational test is not internal agreement alone. It is whether a group can question its own justification for power, including under competitive pressure.
What it means for you
- Test longer workflows. Choose a multi-step agent task with changing conditions. Track completion, recovery, and the cost of each failure over repeated runs.
- Separate capability measures. Report performance by task rather than relying on a broad capability label. Include the conditions under which the system succeeds or fails.
- Write down desired behavior. Ask teams to define good behavior for their use case and explain why it is beneficial. Track unresolved disagreements alongside evaluation results.
- Review escalation paths. Map who can challenge an automated decision or stop deployment. Check whether those routes work when they conflict with delivery pressure.
What to watch
- Whether long-running agent evaluations report security, legal, and competitive risks alongside task completion.
- Whether forecasts distinguish partial AI research automation from self-sustaining improvement.
- Whether labs explain their alignment goals and examine their own influence.
Send this to product and operations leaders setting agent autonomy limits.
Watch the minute where Ngo describes the six-month company scenario
Written by Real Alpha from the episode's transcript and the other moments it links, then checked against the transcript. Quotes are the speaker's exact words.
Minute by minute
- 0:00Company operations demand robustness
Richard Ngo says running a company for six months while keeping it profitable and out of legal trouble could be difficult for AI models. A larger company has a broad attack surface: hackers, rivals or one mistake could trigger legal trouble that shuts it down.
- 1:01Lab advocacy must confront power
Ngo says effective altruism had historically been conciliatory toward AI labs because people feared challenging prestigious, powerful companies. He sees speaking against lab race dynamics as a positive step, while warning that good-versus-bad framing can obscure the power dynamics.
- 4:35Forecasts need more than fast takeoff
Ngo says many people in the field are scared by the pace of progress and may still underestimate it, but struggle to imagine AI companies growing more powerful without recursive self-improvement. The host says claims that AI R&D will soon be fully automated and trigger an intelligence explosion go beyond the evidence; Ngo agrees.
- 7:56A strange future need not mean takeoff
Ngo says the evidence does not justify expecting rapid, extreme self-improvement in the next few years. But he also rejects a normal future: he expects AI to make the next decade much stranger, even if no sudden jump into superintelligence occurs.
- 10:30AGI and superintelligence need distinction
Ngo says people once used AGI as a broad term for very sophisticated AI and often conflated it with superintelligence. He argues the terms describe different things, and says superintelligence itself needs to be unpacked into more precise concepts.
- 12:37Capabilities will remain jagged
Ngo says models are already close to superhuman at some kinds of math and may become superhuman at novel writing reasonably soon. He sees long-term company operations as harder because they require robustness against hackers, rivals and legal risks; reliably running a profitable company for six months could be difficult.
- 14:40Alignment can shape capability access
Ngo says models may have capabilities that people do not know how to elicit. He points to chain-of-thought prompting, which revealed overlooked abilities in early language models after one to two years, and says models may show different capabilities depending on whether they trust the person or organization prompting them.
- 17:21Alignment research needs clearer concepts
Ngo says labels such as alignment are too imprecise to show whether research advances safety or capabilities. He favors clarifying concepts and making explicit arguments for what good AI behavior should look like, citing disagreement over how to interpret Anthropic’s alignment-faking paper and the rationale for Claude’s constitution.
- 22:11Safety work depends on trusted groups
Ngo says small groups need trust and clear communication to coordinate and grow their influence. He argues that early OpenAI had more serious AI discussions than mainstream ML or most of DeepMind, and says this helped OpenAI out-compute DeepMind before Anthropic out-computed OpenAI.
- 25:00Ngo says Anthropic should examine its influence
The host suggests Anthropic fits Ngo’s model of a coordinated group. Ngo counters that groups can cohere around dogmas, including the belief that others would build the technology anyway; he says Anthropic should assess its own influence, including fears he attributes to some Chinese researchers.
- 27:16Ngo says elite pressure narrows debate
Asked about his 2024 support for Trump, Ngo says rationalists’ ties to cultural elites and social pressure can make dissent costly. He argues that AI urgency pushed some to avoid political disagreement for fear of losing influence, a strategy he says impaired clear thinking about AI plans.
Published Nov 4, 2026.
Transcript · 22 min read
MTS, hosted by Theo Jaffee and Sophia Dew, with Richard Ngo.
Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.
the sort of thing that feels more difficult is running a company for long periods of time and when you're running a company especially a larger company you just have this enormous attack surface where you know one mistake uh can be exploited by a bunch of adversaries like hackers your uh your rivals you can mess up and then be in trouble with the legal system in a way that shuts down the entire company uh so all of those things prioritize a certain level of robustness that the models don't currently have. And my sense is that it could be pretty difficult to get the models actually reliable on a task like run a company for six months and make sure it's profitable and doesn't end up in legal trouble.
And we are back. We are live with Richard Ngo, who is an independent AI researcher who worked on AGI safety previously at Google DeepMind and on governance at OpenAI. before leaving to become independent. Richard, welcome to MTS. Thanks for having me. Absolutely. So why don't we talk about, start with, there seem to be a lot of people now in the mainstream EA world recommending resigning from the labs, making things more difficult for the labs. Like Robert Wiblin, for example, from the 80,000 Hours podcast just posted this long video today, actually, about how people have been able to get the labs to change behavior basically through activism.
And you've been talking about this for some time. You've been recommending people resign from labs. Do you think that this is a positive change from the mainstream of EA?
Read the whole transcript
“And not, I think, for a particularly good reason, just because people were somewhat conflict-averse and scared of challenging, prestigious, powerful companies.”
currently writing this sequence of posts on less wrong uh as a retrospective against the failures of the AI alignment movement over the last decade or so. Do you think that the community has responded at all to this? Do you think they have, have you seen any signs of change towards what you would want them to be doing?
“people are jumping further than the evidence warrants towards, you know, strongly expecting a very fast RSI of next couple of years.”
I want to shout out our sponsor, Lovable. You know the app you've been meaning to build, the internal tool, side project, or product you'd otherwise lose a weekend to? With Lovable, you get software you can ship today. That includes an editable codebase you can inspect and change, plus two-way GitHub Sync. Lovable also handles the backend and infrastructure, including managed Postgres, auth, storage, hosting, payments, and more, so you don't have to wire it all up yourself.
And through Lovable's MCP server, you can create and deploy your project directly from the agents you already use. Turn ideas into software people love with Lovable. Lovable.dev. Now, on to the episode.
“it's become apparent that AGI is very different from superintelligence.”
“the sort of thing that feels more difficult is running a company for long periods of time”
We'll continue monitoring right after this message from our sponsors. Scale your startup on Neon. Millions of developers and startups have already chosen Neon for their backend. Start on the free plan or get up to $100,000 in credits for your startup at neon.com slash MTS. Special thanks to our sponsor Kong, the AI connectivity platform. Connect APIs, LLMs, agents, and systems with serious security and governance. konghq.com. Finally prove what your AI investment is worth with Lairdyn. Head to lairdyn.com today to start measuring your AI's bottom line. MongoDB, trusted to run critical applications by 75% of the Fortune 100, the intelligent data platform for the era. Apply today for access to up to $15,000 in Atlas credits, dedicated technical support, and exclusive events and go-to-market opportunities.
MongoDB.com. Chargebee, billing and monetization infrastructure for the AI era. Chargebee.com. so you talk about alignment as a bottleneck on capabilities um this seems to be sort of different from how uh or rather you you talk about alignment as a bottleneck of capabilities almost as a positive thing here am i getting that right whereas elsewhere you've talked about you know the idea of uh prosaic alignment being a bottleneck on capabilities as like uh indication that the entire field of AI alignment has been pursuing the wrong things, right? Yeah. My sense is mostly that, again, we've reached a point where these words are too low resolution.
And so we just have these categories that don't quite make sense. And this is sort of the default you should expect from trying to forecast the future, that you're going to come up with these concepts and they're sort of not really going to be quite right. And over time, the gaps between the concepts and reality, the concepts are going to end up with a bunch of holes in them or the edge case is not going to be very well defined. And so like we have with AGI, right, like whether or not we have AGI already is in many ways a definitional debate. And the definition that people used in the past wasn't typically precise enough for us to actually have a clear resolution to that debate today.
So, yeah, I guess the main point I want to make is that we're going to need to think much more carefully about the kinds of research that are actually valuable and not. And just because something has historically been called alignment doesn't mean very much. That could mean that it's actually one of the things that advances capabilities the most. uh that that all feels very up for grabs especially given that the companies are using these terms in a way that um you know tries to sneak as much as possible under the banner of alignment right
“I don't buy this idea that you need to be in the labs using frontier models in order to do good research.”
movement in general? Obviously, working on alignment research that would produce actual fundamental alignment of the models is a good one. But, you know, there's other components to this.
“my sense is that Anthropic as an organization is not willing to look at its own influence on the world”
So one other question is that in 2024, you are like one of the very few people in the broader rationalist space to publicly back Donald Trump in general politics and specifically, you know, the binary decision on whether to vote for Democrats or Republicans is something that very smart people operating even very similar models of the world can differ on.
like they can come to uh one of these two choices or the other uh what do you think are the cruxes between you and the rest of or most of the rest of the rationalist movement on this in particular my sense is that rationalists in general are cultural elites um or like very closely affiliated with cultural elites and by cultural elites i mean here like you know harvard graduates or like top university graduates and then a sort of class of people that has gone under various names like professional managerial class is one term or class x is another term that's been used to describe the sort of current cultural ruling class of america and it's just difficult to go against your class interests if you're like it's very scary to face repercussions from the people around you and i think you sort of see this with the level of elite hysteria over brexit and over trump's original election where really there was some sense of people being whipped into a frenzy in a way that was hard to, like, wasn't really very grounded in specific things that they were worried about.
It was much more a sense of, like, the elite class is mobilizing against the working class, especially the white working class. And so, like, my basic sense is that, you know, most people who have this kind of intellectual background in the educator class are just going to be held in place by ties of, you know, ties of ideology combined with social pressure. And rationalists were good at breaking out of some of these, for example, by thinking about AI, even when that was seen as like weird, but other pressures are sort of scarier and harder to speak out against. And in some ways, like urgency about AI, I think, made rationalists less willing to think about the political situation. It's like, well, if you've got this big priority, which is AI, then you shouldn't rock the boat too much in other ways because that will lose influence. And unfortunately, it seems to have turned out that this was a bad strategy, that you really needed to be thinking clearly about all this stuff in order to make good plans about the future of AI. Right. Well, we're coming up on time, but it's been very interesting to have you on.
I think you're one of the clearest thinkers on a lot of these issues. There are a few people who are able to sort of escape the kind of groupthink bubble that surrounds many in AI safety. So much respect. I can't wait to read the rest of your sequence on alignment. Thanks so much for coming on MTS. I enjoyed the conversation. Thank you. A huge thanks to MTS sponsors. Blitzy, autonomous software development for enterprise code bases. Ship 5x faster. Blitzy.com. AdQuick, make your brand a billboard. Out-of-home advertising as easy to scale as digital. AdQuick.com. Arena, measuring AI performance in the real world.
Arena.ai. Support for the show comes from VCX, the public ticker for private tech. The U.S. stock market started history's greatest wave of wealth creation. From factory workers in Detroit to farmers in Omaha, anyone could own a piece of the great American companies. But today, our most innovative companies are staying private longer, which means everyday Americans are missing out. Until now. Introducing VCX, a public ticker for private tech. Visit GetVCX.com for more info. That's GetVCX.com. Carefully consider the investment material before investing, including objectives, risk, charges, and expenses.
This and other information can be found in the Innovations Fund Prospectus at GetVCX.com. This is a paid sponsorship.
