# AI Models Will Learn at Test Time by Next Year | Rohan Anil

Markdown copy of https://realalpha.app/v/rVMy8MSBKZo/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

Rohan Anil says Core Automation is building models that learn from their mistakes at test time, with a goal of making it work next year.

- Show: MTS
- Hosts: [Theo Jaffee](https://realalpha.app/people/theo-jaffee/) & [Sophia Dew](https://realalpha.app/people/sophia-dew/)
- Guest: [Rohan Anil](https://realalpha.app/people/rohan-anil/) is cofounder of Core Automation and former Anthropic pre-training researcher.
- Published: Nov 5, 2026
- Length: 22 min

## The best minutes

- [15:37](https://realalpha.app/v/rVMy8MSBKZo/?t=937) [Test-time learning may be a bigger bottleneck than compute](https://realalpha.app/s/test-time-learning-may-be-a-bigger-bottleneck-than-compute/) (2 min)
- [1:47](https://realalpha.app/v/rVMy8MSBKZo/?t=107) [Compute islands could make scattered GPUs useful for frontier training](https://realalpha.app/s/compute-islands-could-make-scattered-gpus-useful-for/) (2 min)
- [9:20](https://realalpha.app/v/rVMy8MSBKZo/?t=560) [Butterfly mixing lets training tolerate asynchronous GPU updates](https://realalpha.app/s/butterfly-mixing-lets-training-tolerate-asynchronous-gpu/) (2 min)

## Also worth watching

- [13:22](https://realalpha.app/v/rVMy8MSBKZo/?t=802) [Distributed training could make idle GPU clusters usable](https://realalpha.app/s/distributed-training-could-make-idle-gpu-clusters-usable/) (2 min)

Skip the distributed-PC and chip-design discussion, the space-training section, and the alignment-risk close and sponsor reads.

## Summary

Rohan Anil is a former Anthropic pre-training researcher and cofounder of Core Automation. He says the company is developing architecture and training algorithms for models to learn at test time, and has explored training across distributed compute islands.

Anil argues that models’ inability to learn quickly from their own mistakes is a key bottleneck, and says Core Automation is working on test-time learning with a goal of making it work next year. He says compute islands can combine GPUs in different locations to train frontier models, and that asynchronous training methods can make this practical. He sees chip design as a possible direction for labs with the necessary talent, and says distributed training could help labs use otherwise idle compute. He also says continual learning raises alignment and autonomy concerns, though it could help models defend systems as they learn.

### Minute by minute

- [0:49](https://realalpha.app/v/rVMy8MSBKZo/?t=49) **Anil says test-time learning is next.** Anil says he left Anthropic to work on what he sees as the next frontier: models that learn at test time. He argues that scaling has produced agents, but models still need a way to become more useful by learning from experience.
- [1:47](https://realalpha.app/v/rVMy8MSBKZo/?t=107) **Compute islands widen access to frontier training.** Anil describes compute islands as GPUs distributed across locations rather than colocated in one data center. He says Core Automation’s approach could use different chips and configurations, and that its experiments produced results comparable to or slightly better than its scaling baseline.
- [4:02](https://realalpha.app/v/rVMy8MSBKZo/?t=242) **Distributed GPUs could help smaller labs.** Anil says training on CPUs is possible, but frontier training would be inefficient on weaker hardware because of bottlenecks such as memory and latency. His focus is combining GPUs across data centers; he says this could let a lab with enough compute compete among leading models.
- [7:45](https://realalpha.app/v/rVMy8MSBKZo/?t=465) **Labs may design their own chips.** Anil says labs could move into chip design if they have people with the skills to work across the stack. He notes that models are helping accelerate model development, but says chip design still requires specialized domain expertise.
- [9:20](https://realalpha.app/v/rVMy8MSBKZo/?t=560) **Butterfly mixing tolerates asynchronous training.** Anil explains that butterfly mixing lets machines work on different fragments without coordinating every step, then exchange information over three steps. He says a 32-billion-parameter mixture-of-experts Transformer learned representations robust to the resulting noise and reached the quality of distributed training.
- [11:28](https://realalpha.app/v/rVMy8MSBKZo/?t=688) **Distributed training could reach space.** Anil says the algorithms for distributed training could support training in space, while the harder questions are physical. He points to Google’s Suncatcher project running inference on TPUs with Gemma 4, and says StarCloud trained nanoGPT on a satellite using one H100.
- [13:22](https://realalpha.app/v/rVMy8MSBKZo/?t=802) **Aggregation could unlock idle compute.** Anil says compute markets appear inefficient, with some short-term capacity potentially sitting idle while buyers seek large colocated clusters. He argues that training across compute islands could make scattered GPUs useful for frontier runs, including 10,000 separate islands.
- [15:37](https://realalpha.app/v/rVMy8MSBKZo/?t=937) **Learning from mistakes is a bottleneck.** Anil says models are jagged because they lack mechanisms to learn from their own mistakes quickly. He argues that labs currently need repeated pre-training and reinforcement-learning cycles, and that continual learning could let a model adapt as a company evolves.
- [17:48](https://realalpha.app/v/rVMy8MSBKZo/?t=1068) **Continual learning brings new alignment risks.** Anil says test-time learning raises questions about what models should learn, what they should be shown, and autonomy. He also sees possible defensive uses, such as models learning to secure systems, while saying the risks need more study.
- [19:23](https://realalpha.app/v/rVMy8MSBKZo/?t=1163) **Anil targets test-time learning next year.** Anil says Core Automation will try to make test-time learning work next year. He encourages the hosts to monitor the effort.

## Transcript

MTS, hosted by Theo Jaffee and Sophia Dew, with Rohan Anil.

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[0:00](https://realalpha.app/v/rVMy8MSBKZo/?t=0) The reason it's jagged is that we don't have mechanism in the architecture and optimization for models to be able to learn from its own mistakes much quicker. And today we have to wait for labs to do the pre-training and RL cycle every K-months with the right environments and pre-training data to solve those tasks. We are designing the algorithms that work at test times. We will try our best and I think we can get it done next year. And we are back. We are live in the studio with Rohan Anil, who's co-founder of Core Automation. We've had Jerry Torek, the other co-founder of Core Automation, on before. He was great. This is one of my favorite interviews ever. I think this one might even be better. So Rohan, welcome to MTS. Thank you for having me and I'll try. So I'm curious to start. You left Anthropic to join or start Core Auto. No one ever leaves Anthropic. What made you leave Anthropic?

[1:01](https://realalpha.app/v/rVMy8MSBKZo/?t=61) Oh, I think it's a meme that people don't leave Anthropic. I think Anthropic is one of the best places to do research. I think I really love the pre-training team that I worked on. I think it's just, I think people have different motivations in life. Mine was to focus on the next frontier, which I think we're building at Koro. We have had models that when you scaled up, brought us agents and brought us the world that we have today. What is the next piece of technology that can make it extremely useful in a way that changes our lives? I mean, we're all going to work. We're still using the AI. We are actually working even harder than we worked before.

[1:44](https://realalpha.app/v/rVMy8MSBKZo/?t=104) So what is missing in these models? And it's the ability to learn at test time. And that's something that we are working in. Yeah. So tell us about, you recently wrote the Hitchhiker's Guide to Training on Compute Islands. For those unfamiliar, what are compute islands and how do we maximally utilize compute? Yeah, so at a very high level, either you have compute to train a frontier model or you don't. I think from training perspective, the most valuable computation is frontier training. And if you can get to frontier, people care. The top two models matter and everything else don't matter as much or have a very limited shelf life.

[2:24](https://realalpha.app/v/rVMy8MSBKZo/?t=144) when we started the company it was quite clear to accrue a lot of compute it's it's not just the money it's it's about co-location it's about data centers it's able to have that much power and physical constraints come in the way of delivery of many of these things but on the other hand we were getting a sufficient number of like thousands of or hundreds of gb300s hundreds

[2:47](https://realalpha.app/v/rVMy8MSBKZo/?t=167) **Rohan Anil:** of p300s or thousands of them at different locations that you could get and if you had a training algorithm that can actually use them to get to frontier then you can unlock compute wherever it is and you can also get to frontier much quicker in some sense this is as i put it

[3:06](https://realalpha.app/v/rVMy8MSBKZo/?t=186) in the blog post it's not new like people who worked in this field have been working on this entire another set of area of this local SGD federated learning and very it's it's not as popular but there is like an emerging area of training in a distributed set of islands or compute islands yeah and if you could do that well as well as perhaps training otherwise then it flips the question of like you could use any type of chips in our experiment we used two types one is grace blackwell and the other one is just blackwell so one is 70 and we link 72 and the other one is and we link eight they use completely different parallelism they have slightly different numerics due to the parallelism configuration completely different batch size and we what we provide our main result is we get similar a slightly better result when we scale up on a very efficient frontier training, scaling, scaling law.

[4:05](https://realalpha.app/v/rVMy8MSBKZo/?t=245) Yeah, so. Yeah, what happened to the old dream of training AI models using normal compute in people's computers? I remember this was a thing that some people were working on back in like 2022. Maybe it just became like unfeasible due to the scale of the models that are currently being trained. But like, you know, why wouldn't it be possible in theory, In the same way that people using their ordinary computers in a distributed way can be used for seeding torrents, why can't they be used to train models? I think I would say there are still folks working in this area. There are dreamers in this world who are pushing on it.

[4:44](https://realalpha.app/v/rVMy8MSBKZo/?t=284) I think I've cited them in the blog. But at the same time, I think there are two problems. I think in general, I think this is possible to train on CPUs, but if you want to train frontier, you are training, at least right now, you're training some sufficient number of parameters, some number of flops, which means that we would be, it would not be as efficient to train on these worse hardware because we will get into Amdahl's bottleneck. it could be memory, it could be latency, and we don't, or at least the work that I'm putting out is not addressing that. This one is addressing this narrow case of, can we actually train frontier on, say, I get 10 different data centers with thousands of GPUs? Can I get to the current frontier in terms of, say, scaling pre-training? The answer seems yes, and you can with clever algorithmic improvements. Yeah. So what does this mean for Neolabs being able to utilize compute to maybe like, you know, compete with Frontier Labs or maybe having the same, like having some efficiency advantage over the Frontier Labs? I think so like at least at Corodo, we don't direct, I mean, there's a Frontier game and we are truly focused on novel new algorithms that to get us to a different axis like continual learning but if you were in your lab that's

[6:09](https://realalpha.app/v/rVMy8MSBKZo/?t=369) trying to play the same game of pre-training and rl this lets you allow allow you to use the flops more efficiently and say get into that same ring of like top three in artificial analysis as long as you can get those compute quickly yeah and i think that itself could be useful if you were verticalizing your company to say chip design or something else and so this seems like a feasible approach yeah so for continual learning specifically um how have you guys thought about like the physical like utilizing hardware or how does that differ i think um it's maybe too early to talk about the research progress but the ones that we have seen sufficiently we are designing the algorithms that work at test time so we are actually changing both the architecture and the training algorithm together.

[7:01](https://realalpha.app/v/rVMy8MSBKZo/?t=421) So it is an efficiency play as well. So we are, as Jerry puts it, removing the bottlenecks that exist in the Transformer. Hmm, interesting. I want to shout out our sponsor, Lovable. You know, the app you've been meaning to build, the internal tool, side project, or product you'd otherwise lose a weekend to? With Lovable, you get software you can ship today. That includes an editable code base you can inspect and change, plus two-way GitHub sync. Lovable also handles the backend and infrastructure, including managed Postgres, auth, storage, hosting, payments, and more, so you don't have to wire it all up yourself.

[7:36](https://realalpha.app/v/rVMy8MSBKZo/?t=456) And through Lovable's MCP server, you can create and deploy your project directly from the agents you already use. Turn ideas into software people love with Lovable. Lovable.dev. Now, on to the episode. You talked about labs verticalizing into chip design. We've seen OpenAI do this with Jalapeno. We've seen Anthropics start to do this. Do you think every lab It's going to eventually design their own chips. Yeah. I think there's, uh, I mean, like we say something at Coroto, everything is a skill issue. So in some sense, uh, and it's in a way, in a way, if you can get the sufficiently talented people who can go full stack, I think it is inevitable that these models just want to learn.

[8:27](https://realalpha.app/v/rVMy8MSBKZo/?t=507) And these models, we are able to now use models to accelerate the process of developing models. And it's been, hey, so the way you develop models today is very different from, say, four years ago. Things have substantially changed what you could do with the compute you have and with a fewer number of people. So I could see this happening as more and more folks can get to frontier on, like, vertical domains. it's still going to be hard because you still need the domain expertise of having designed a chip having knowing how like the whole process of building chips and so it is possible to do that but it's going to be it's going to be hard but now at least the components for example the one that we're talking about is training using computer islands to scale up the flops you can get seems possible so it's like foundational in terms of what improving the uh possible possibilities for people who are able to capture it.

[9:24](https://realalpha.app/v/rVMy8MSBKZo/?t=564) Yeah. So you talk about revisiting butterfly mixing. What is butterfly mixing? This is a very old idea before neural networks. The idea here is in a typical neural network training on, say, NVIDIA chips, we have this all-reduction, which is to communicate all the gradients across all the GPUs and have single states. So say summing all of them up across all the GPU state, what we find is that you can do this across many steps and you can have multiple different machines working on different fragments at a time. so they are not in coordination and within three steps you exchange all of them but they're still

[10:12](https://realalpha.app/v/rVMy8MSBKZo/?t=612) **Rohan Anil:** not caught they're still working on slightly different pieces of information but that's sufficient for the model to attain the same quality as if you had done distributed training

[10:23](https://realalpha.app/v/rVMy8MSBKZo/?t=623) so what we are finding is the neural networks that we train in this case it was a 32 billion transformer. It was a mixtures of expert model. It had 32x persity. We were able to see the model was able to use, learn representations that were robust to what this is causing, which is some type of noise. In some sense, for me, this is not surprising because I started working on neural networks all the way back when we had CPUs. And I was training neural networks for Google for search on CPUs using asynchronous training. And then we actually, at that point, thought that we had to put asynchronous training into the chips.

[11:08](https://realalpha.app/v/rVMy8MSBKZo/?t=668) And there was this mad dash, like, will synchronous training actually work? And I saw the synchronous training paradigm take over and just work and scale up through GPT-3, GPT-4, Gemini, entropic models. And now we are seeing that, hey, actually with some clever methods, like we use second order methods in this case, and a few other tricks, we are able to like make use of your synchronicity to improve efficiency. In this case, like efficient use of compute islands to build a model. So what will it take to actually train a like true frontier scale model fully in space, which, you know, maybe that'll become necessary in the next couple of years for some reason.

[11:51](https://realalpha.app/v/rVMy8MSBKZo/?t=711) This is where, when I was writing this article, like you want to look at what the most ambitious thing of your research could be? And I do think like while at least personally I've not built satellites, nor do I understand thermodynamics and things like what is required to build it. But you can see Google make progress. For example, the Suncatcher project, they were able to run inference on TPUs on a Gemma 4 model. And I think also I read that StarCloud was able to train nano GPT on a satellite some time ago. So it seems like it's nano, right? Wasn't that on like 1H100? 1H100. Everything starts small. So it is a question that you could, if, I think Ilan was mentioning on hex that it could be possible.

[12:41](https://realalpha.app/v/rVMy8MSBKZo/?t=761) And if the science and the physics and the reality is that this could be possible, then this makes a lot of sense from an algorithmic perspective. and this article is saying well the algorithms exist to train in a distributed manner and now we have to solve all the much more harder part which is the physical constraints and so in some sense like i uh it would be if something like that is feasible it's worth making progress on and one of the reasons to write this article was to also mention that the algorithms do exist Now it's just a question of harnessing it into something that is feasible in hopefully our lifetimes, maybe in our kids' lifetimes, right?

[13:25](https://realalpha.app/v/rVMy8MSBKZo/?t=805) Yeah, yeah. I'm also curious how you see compute markets evolving or how you experience that so far. You talk about how there are just these short-term unused capacity compute that are now being sold by companies like SF Compute. Yeah, so I think there are many short-term and I do notice that the market doesn't look that efficient. So this was us mentioning, hey, there could be an efficiency because perhaps there's more value and aggregating a bunch of compute, like the value of the compute goes higher if it is for a frontier run versus if it's for some other like a low value inference. So you could, I think there could be efficiency in the market having different type of training technology.

[14:12](https://realalpha.app/v/rVMy8MSBKZo/?t=852) I do see there is much harder to get tens of thousands of Grace Blackwells than it is to get a few hundred or a few thousand. So I do see maybe these chips are sitting idle till it gets sold. And perhaps the people who demand a lot more are looking for co-location because of an algorithmic constraint and a practical constraint. If you ask me, do you want 10,000 Blackwells? Yeah, sure. I'll take it. But if you tell me, like, will you take 10,000 separate computer islands?

[14:45](https://realalpha.app/v/rVMy8MSBKZo/?t=885) **Rohan Anil:** Without this technology, I wouldn't. But with this, I would be like, yeah, sure.

[14:50](https://realalpha.app/v/rVMy8MSBKZo/?t=890) I'll work through the infrastructure pain to train algorithmic pain. We don't feel because we work on these algorithms day and night. So we can train frontier with that many GPUs. So it's a question of enablement. and usually like it's much harder for us to invert the problem so this blog post is one way to get people's attention of like hey there is a new a new emerging technology that could be used it could also look like multiple organizations come together to build something bigger if they're all aligned on the same values or the same utility function because compute is spread out right so if If you look at it from a holistic perspective, how do you maximally utilize compute to have productive value to the world?

[15:36](https://realalpha.app/v/rVMy8MSBKZo/?t=936) Perhaps there are angles to this that I'm very hopeful for. Yeah, totally. So you also talk in this piece a lot about the bottlenecks to deployment of compute because we don't have like grid equipment. We don't have the power. We don't have the land. Maybe there will also be new political bottlenecks. um how much to these all in all affect your timelines to AGI, like real AGI? I mean, the reason I'm at CoraO and I'm working on the problems is that I think there is fundamental, like these models are special. I love Astra, I love my opus, I love my fable. But I mean, I think everybody who's watching and both of you probably love them too in ways, but you probably hate them too in other ways.

[16:28](https://realalpha.app/v/rVMy8MSBKZo/?t=988) the number of times I've had incidents of rage when the models gaslight me, not able to do something simple, like when it can solve naivier stokes on one day and do something dumb on the

[16:40](https://realalpha.app/v/rVMy8MSBKZo/?t=1000) **Rohan Anil:** other day. It's very jagged. I think it's something Karpathy coined. And the reason it's jagged is that we don't have mechanism in the architecture and optimization for models to be able to learn from its own mistakes much quicker. And today we have to wait for labs to do the

[16:57](https://realalpha.app/v/rVMy8MSBKZo/?t=1017) pre-training an RL cycle every K months with the right environments and pre-training data to solve those tasks. So I feel like that is a limiting factor, at least personally. I feel like beyond compute, there is an algorithmic limiting factor for us to make this useful in a way that I think could get us to like, we exist and we are able to use this technology to do something bigger. If companies could operate much more effectively, if there was one piece of model that can learn as a company evolves today the best we can do is compacted tokens in different forms of different types of memory and we would like to solve this at the fundamental layer and then build products on top so i do think i think i hedged the question to the answer by saying there is i think even without the data center power and um grid acumen there is some other algorithmic bottleneck that exist as well. Yeah. When we do get to that continual learning, or at least the first primitive type of continual learning, what do you think is the alignment risk or how does that differ from the current alignment risk? I think it's a bigger topic and needs to be carefully studied. One thing is to talk about if you have models that are able to learn at test

[18:17](https://realalpha.app/v/rVMy8MSBKZo/?t=1097) time what it should learn and what it should be shown and what is an interesting question of autonomy and so there is more possible areas of risk involved in this technology just like today's technology for example um increasing cyber capabilities have created this world that we live in in the last week there has been companies getting hacked all over the place because cves i think somebody's statistic here was there were more cves being generated because there's more white coded slop most of the code and even with the best defensive models the like you're able to fix it even on software that we depend on like that software we use here software we use at the company is much harder so i think the risks are involved but on the other hand i do think something like continual learning may act as a defense mechanism if uh if models are able to secure systems as it learns on the fly it could be used as in a defensive way and And this is why I think this topic does demand a lot more than I could possibly talk about today.

[19:23](https://realalpha.app/v/rVMy8MSBKZo/?t=1163) Yeah, absolutely. Well, we're coming up on time, but this has been super interesting. You know, I can't wait for the models to be able to continually learn so I can actually teach them how to do things that they're not good at and then have them become good at it. So very excited for this to become a reality. We will try our best, and I think we can get it done next year. Wow. Wow, a bold. Bold. That's the block we're going to be monitoring. Yeah, thanks so much. You should monitor this. Thanks so much, Rohan. Thank you. Thanks so much. A huge thanks to MTS sponsors. ElevenLabs, AI that communicates at human level across every channel and modality.

[19:56](https://realalpha.app/v/rVMy8MSBKZo/?t=1196) 11labs.io slash MTS. Scale your startup on Neon. Millions of developers and startups have already chosen Neon for their backend. Start on the free plan or get up to $100,000 in credits for your startup at neon.com slash MTS. Special thanks to our sponsor Kong, the AI connectivity platform. Connect APIs, LLMs, agents, and systems with serious security and governance. konghq.com. Finally prove what your AI investment is worth with Lairdyn. Head to lairdyn.com today to start measuring your AI's bottom line. MongoDB, trusted to run critical applications by 75% of the Fortune 100, the intelligent data platform for the era. Apply today for access to up to $15,000 in Atlas credits, dedicated technical support, and exclusive events and go-to-market opportunities. MongoDB.com. Chargebee, billing and monetization infrastructure for the AI era.

[20:53](https://realalpha.app/v/rVMy8MSBKZo/?t=1253) chargeb.com blitzy autonomous software development for enterprise code bases ship 5x faster blitzy.com adquick make your brand a billboard out-of-home advertising as easy to scale as digital adquick.com arena measuring ai performance in the real world arena.ai support for the show comes from vcx the public ticker for private tech the u.s stock market started history's greatest wave of wealth creation. From factory workers in Detroit to farmers in Omaha, anyone could own a piece of the great American companies. But today, our most innovative companies are staying private longer, which means everyday Americans are missing out. Until now. Introducing VCX, a public ticker for private tech. Visit getvcx.com for more info.

[21:42](https://realalpha.app/v/rVMy8MSBKZo/?t=1302) That's getvcx.com. Carefully consider the investment material before investing, including objectives, risk, charges, and expenses. This and other information can be found in the Innovations Fund prospectus at getvcx.com. This is a paid sponsorship.