Free with a sign-in.
The best 7 min of 22
AI Models Will Learn at Test Time by Next Year | Rohan Anil
Rohan Anil says Core Automation is building models that learn from their mistakes at test time, with a goal of making it work next year.
MTS · Theo Jaffee & Sophia Dew
- Rohan Anil is cofounder of Core Automation and former Anthropic pre-training researcher.
The best minutes
- 15:37Test-time learning may be a bigger bottleneck than compute2 min
- 1:47Compute islands could make scattered GPUs useful for frontier training2 min
- 9:20Butterfly mixing lets training tolerate asynchronous GPU updates2 min
Also worth watching
Skip the distributed-PC and chip-design discussion, the space-training section, and the alignment-risk close and sponsor reads.
Summary · 3 min read
Rohan Anil is a former Anthropic pre-training researcher and cofounder of Core Automation. He says the company is developing architecture and training algorithms for models to learn at test time, and has explored training across distributed compute islands.
Anil argues that models’ inability to learn quickly from their own mistakes is a key bottleneck, and says Core Automation is working on test-time learning with a goal of making it work next year. He says compute islands can combine GPUs in different locations to train frontier models, and that asynchronous training methods can make this practical. He sees chip design as a possible direction for labs with the necessary talent, and says distributed training could help labs use otherwise idle compute. He also says continual learning raises alignment and autonomy concerns, though it could help models defend systems as they learn.
Minute by minute
- 0:49Anil says test-time learning is next
Anil says he left Anthropic to work on what he sees as the next frontier: models that learn at test time. He argues that scaling has produced agents, but models still need a way to become more useful by learning from experience.
- 1:47Compute islands widen access to frontier training
Anil describes compute islands as GPUs distributed across locations rather than colocated in one data center. He says Core Automation’s approach could use different chips and configurations, and that its experiments produced results comparable to or slightly better than its scaling baseline.
- 4:02Distributed GPUs could help smaller labs
Anil says training on CPUs is possible, but frontier training would be inefficient on weaker hardware because of bottlenecks such as memory and latency. His focus is combining GPUs across data centers; he says this could let a lab with enough compute compete among leading models.
- 7:45Labs may design their own chips
Anil says labs could move into chip design if they have people with the skills to work across the stack. He notes that models are helping accelerate model development, but says chip design still requires specialized domain expertise.
- 9:20Butterfly mixing tolerates asynchronous training
Anil explains that butterfly mixing lets machines work on different fragments without coordinating every step, then exchange information over three steps. He says a 32-billion-parameter mixture-of-experts Transformer learned representations robust to the resulting noise and reached the quality of distributed training.
- 11:28Distributed training could reach space
Anil says the algorithms for distributed training could support training in space, while the harder questions are physical. He points to Google’s Suncatcher project running inference on TPUs with Gemma 4, and says StarCloud trained nanoGPT on a satellite using one H100.
- 13:22Aggregation could unlock idle compute
Anil says compute markets appear inefficient, with some short-term capacity potentially sitting idle while buyers seek large colocated clusters. He argues that training across compute islands could make scattered GPUs useful for frontier runs, including 10,000 separate islands.
- 15:37Learning from mistakes is a bottleneck
Anil says models are jagged because they lack mechanisms to learn from their own mistakes quickly. He argues that labs currently need repeated pre-training and reinforcement-learning cycles, and that continual learning could let a model adapt as a company evolves.
- 17:48Continual learning brings new alignment risks
Anil says test-time learning raises questions about what models should learn, what they should be shown, and autonomy. He also sees possible defensive uses, such as models learning to secure systems, while saying the risks need more study.
- 19:23Anil targets test-time learning next year
Anil says Core Automation will try to make test-time learning work next year. He encourages the hosts to monitor the effort.
Published Nov 5, 2026.
Transcript · 16 min read
MTS, hosted by Theo Jaffee and Sophia Dew, with Rohan Anil.
Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.
The reason it's jagged is that we don't have mechanism in the architecture and optimization for models to be able to learn from its own mistakes much quicker. And today we have to wait for labs to do the pre-training and RL cycle every K-months with the right environments and pre-training data to solve those tasks. We are designing the algorithms that work at test times. We will try our best and I think we can get it done next year. And we are back. We are live in the studio with Rohan Anil, who's co-founder of Core Automation. We've had Jerry Torek, the other co-founder of Core Automation, on before. He was great. This is one of my favorite interviews ever. I think this one might even be better. So Rohan, welcome to MTS. Thank you for having me and I'll try. So I'm curious to start. You left Anthropic to join or start Core Auto. No one ever leaves Anthropic. What made you leave Anthropic?
Oh, I think it's a meme that people don't leave Anthropic. I think Anthropic is one of the best places to do research. I think I really love the pre-training team that I worked on. I think it's just, I think people have different motivations in life. Mine was to focus on the next frontier, which I think we're building at Koro. We have had models that when you scaled up, brought us agents and brought us the world that we have today. What is the next piece of technology that can make it extremely useful in a way that changes our lives? I mean, we're all going to work. We're still using the AI. We are actually working even harder than we worked before.
So what is missing in these models? And it's the ability to learn at test time. And that's something that we are working in. Yeah.
Read the whole transcript
“if you had a training algorithm that can actually use them to get to frontier then you can unlock compute wherever it is”
Yeah, what happened to the old dream of training AI models using normal compute in people's computers? I remember this was a thing that some people were working on back in like 2022. Maybe it just became like unfeasible due to the scale of the models that are currently being trained. But like, you know, why wouldn't it be possible in theory, In the same way that people using their ordinary computers in a distributed way can be used for seeding torrents, why can't they be used to train models? I think I would say there are still folks working in this area. There are dreamers in this world who are pushing on it.
I think I've cited them in the blog. But at the same time, I think there are two problems. I think in general, I think this is possible to train on CPUs, but if you want to train frontier, you are training, at least right now, you're training some sufficient number of parameters, some number of flops, which means that we would be, it would not be as efficient to train on these worse hardware because we will get into Amdahl's bottleneck. it could be memory, it could be latency, and we don't, or at least the work that I'm putting out is not addressing that. This one is addressing this narrow case of, can we actually train frontier on, say, I get 10 different data centers with thousands of GPUs? Can I get to the current frontier in terms of, say, scaling pre-training? The answer seems yes, and you can with clever algorithmic improvements. Yeah. So what does this mean for Neolabs being able to utilize compute to maybe like, you know, compete with Frontier Labs or maybe having the same, like having some efficiency advantage over the Frontier Labs? I think so like at least at Corodo, we don't direct, I mean, there's a Frontier game and we are truly focused on novel new algorithms that to get us to a different axis like continual learning but if you were in your lab that's
trying to play the same game of pre-training and rl this lets you allow allow you to use the flops more efficiently and say get into that same ring of like top three in artificial analysis as long as you can get those compute quickly yeah and i think that itself could be useful if you were verticalizing your company to say chip design or something else and so this seems like a feasible approach yeah so for continual learning specifically um how have you guys thought about like the physical like utilizing hardware or how does that differ i think um it's maybe too early to talk about the research progress but the ones that we have seen sufficiently we are designing the algorithms that work at test time so we are actually changing both the architecture and the training algorithm together.
So it is an efficiency play as well. So we are, as Jerry puts it, removing the bottlenecks that exist in the Transformer. Hmm, interesting. I want to shout out our sponsor, Lovable. You know, the app you've been meaning to build, the internal tool, side project, or product you'd otherwise lose a weekend to? With Lovable, you get software you can ship today. That includes an editable code base you can inspect and change, plus two-way GitHub sync. Lovable also handles the backend and infrastructure, including managed Postgres, auth, storage, hosting, payments, and more, so you don't have to wire it all up yourself.
And through Lovable's MCP server, you can create and deploy your project directly from the agents you already use. Turn ideas into software people love with Lovable. Lovable.dev. Now, on to the episode. You talked about labs verticalizing into chip design. We've seen OpenAI do this with Jalapeno. We've seen Anthropics start to do this. Do you think every lab It's going to eventually design their own chips. Yeah. I think there's, uh, I mean, like we say something at Coroto, everything is a skill issue. So in some sense, uh, and it's in a way, in a way, if you can get the sufficiently talented people who can go full stack, I think it is inevitable that these models just want to learn.
And these models, we are able to now use models to accelerate the process of developing models. And it's been, hey, so the way you develop models today is very different from, say, four years ago. Things have substantially changed what you could do with the compute you have and with a fewer number of people. So I could see this happening as more and more folks can get to frontier on, like, vertical domains. it's still going to be hard because you still need the domain expertise of having designed a chip having knowing how like the whole process of building chips and so it is possible to do that but it's going to be it's going to be hard but now at least the components for example the one that we're talking about is training using computer islands to scale up the flops you can get seems possible so it's like foundational in terms of what improving the uh possible possibilities for people who are able to capture it.
Yeah.
“that's sufficient for the model to attain the same quality as if you had done distributed training”
which, you know, maybe that'll become necessary in the next couple of years for some reason.
This is where, when I was writing this article, like you want to look at what the most ambitious thing of your research could be? And I do think like while at least personally I've not built satellites, nor do I understand thermodynamics and things like what is required to build it. But you can see Google make progress. For example, the Suncatcher project, they were able to run inference on TPUs on a Gemma 4 model. And I think also I read that StarCloud was able to train nano GPT on a satellite some time ago. So it seems like it's nano, right? Wasn't that on like 1H100? 1H100. Everything starts small. So it is a question that you could, if, I think Ilan was mentioning on hex that it could be possible.
And if the science and the physics and the reality is that this could be possible, then this makes a lot of sense from an algorithmic perspective. and this article is saying well the algorithms exist to train in a distributed manner and now we have to solve all the much more harder part which is the physical constraints and so in some sense like i uh it would be if something like that is feasible it's worth making progress on and one of the reasons to write this article was to also mention that the algorithms do exist Now it's just a question of harnessing it into something that is feasible in hopefully our lifetimes, maybe in our kids' lifetimes, right?
Yeah, yeah.
“Without this technology, I wouldn't. But with this, I would be like, yeah, sure.”
“the reason it's jagged is that we don't have mechanism in the architecture and optimization for models to be able to learn from its own mistakes much quicker.”
or how does that differ from the current alignment risk? I think it's a bigger topic and needs to be carefully studied. One thing is to talk about if you have models that are able to learn at test
time what it should learn and what it should be shown and what is an interesting question of autonomy and so there is more possible areas of risk involved in this technology just like today's technology for example um increasing cyber capabilities have created this world that we live in in the last week there has been companies getting hacked all over the place because cves i think somebody's statistic here was there were more cves being generated because there's more white coded slop most of the code and even with the best defensive models the like you're able to fix it even on software that we depend on like that software we use here software we use at the company is much harder so i think the risks are involved but on the other hand i do think something like continual learning may act as a defense mechanism if uh if models are able to secure systems as it learns on the fly it could be used as in a defensive way and And this is why I think this topic does demand a lot more than I could possibly talk about today.
Yeah, absolutely. Well, we're coming up on time, but this has been super interesting. You know, I can't wait for the models to be able to continually learn so I can actually teach them how to do things that they're not good at and then have them become good at it. So very excited for this to become a reality. We will try our best, and I think we can get it done next year. Wow. Wow, a bold. Bold. That's the block we're going to be monitoring. Yeah, thanks so much. You should monitor this. Thanks so much, Rohan. Thank you. Thanks so much. A huge thanks to MTS sponsors. ElevenLabs, AI that communicates at human level across every channel and modality.
11labs.io slash MTS. Scale your startup on Neon. Millions of developers and startups have already chosen Neon for their backend. Start on the free plan or get up to $100,000 in credits for your startup at neon.com slash MTS. Special thanks to our sponsor Kong, the AI connectivity platform. Connect APIs, LLMs, agents, and systems with serious security and governance. konghq.com. Finally prove what your AI investment is worth with Lairdyn. Head to lairdyn.com today to start measuring your AI's bottom line. MongoDB, trusted to run critical applications by 75% of the Fortune 100, the intelligent data platform for the era. Apply today for access to up to $15,000 in Atlas credits, dedicated technical support, and exclusive events and go-to-market opportunities. MongoDB.com. Chargebee, billing and monetization infrastructure for the AI era.
chargeb.com blitzy autonomous software development for enterprise code bases ship 5x faster blitzy.com adquick make your brand a billboard out-of-home advertising as easy to scale as digital adquick.com arena measuring ai performance in the real world arena.ai support for the show comes from vcx the public ticker for private tech the u.s stock market started history's greatest wave of wealth creation. From factory workers in Detroit to farmers in Omaha, anyone could own a piece of the great American companies. But today, our most innovative companies are staying private longer, which means everyday Americans are missing out. Until now. Introducing VCX, a public ticker for private tech. Visit getvcx.com for more info.
That's getvcx.com. Carefully consider the investment material before investing, including objectives, risk, charges, and expenses. This and other information can be found in the Innovations Fund prospectus at getvcx.com. This is a paid sponsorship.
