# Robots need local models for fast, reliable control

Markdown copy of https://realalpha.app/s/robots-need-local-models-for-fast-reliable-control/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Neil Nie](https://realalpha.app/builders/neil-nie/) · Neil Nie is CEO of Verne Robotics. · Builds general-purpose robots for businesses and researches robot learning as Verne's co-founder and CEO.
- Host: [Sophia Dew](https://realalpha.app/people/sophia-dew/) · Host, MTS
- Show: MTS
- Episode: [Robots Will Train Mostly in Simulation by 2027 | Neil Nie](https://realalpha.app/v/zONxIHfiY9k/), Nov 15, 2026
- Moment: 8:14 to 13:51 of the episode (6 min), playing on https://realalpha.app/s/robots-need-local-models-for-fast-reliable-control/; the episode from there: https://realalpha.app/v/zONxIHfiY9k/?t=494

## Quote

> “So fun fact, if you play baseball, right, you know the human reaction time is about **200 milliseconds**, right?”
> — [Neil Nie](https://realalpha.app/builders/neil-nie/) at [10:05](https://realalpha.app/v/zONxIHfiY9k/?t=605) · MTS

## Takeaways

1. Robot policies may need to produce actions five times per second.
2. Cloud latency and Wi-Fi jitter can undermine safety-critical control.
3. A local model can handle immediate responses while cloud models tackle harder tasks.

## About this segment

Neil Nie describes why safety-critical robot policies must respond locally, while slower, more capable models can run in the cloud. The hybrid setup balances low latency with broader reasoning.

From “Robots Will Train Mostly in Simulation by 2027 | Neil Nie”, Nov 15, 2026. [Watch the full episode](https://realalpha.app/v/zONxIHfiY9k/)

## Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[8:18](https://realalpha.app/v/zONxIHfiY9k/?t=498) Lovable.dev. Now, on to the episode. One of the things I wanted to ask about, what does this mean? I mean, right now, when we think about privacy, and being able to do everything locally. In the software world, of course, it's important, but not something crucial. In the robotics world, it's a make or break because you can't always go to the cloud. If there's a little bit of latency and you're running a full humanoid and there's not great Wi-Fi in a corner of the house and a humanoid breaks, that kills the entire product.

[8:53](https://realalpha.app/v/zONxIHfiY9k/?t=533) And so does that, paired with open source, have a big factor as well? Yeah, totally. We think that these System 0 and System 1 models are meant to be running locally on device, right? And we think that that's exactly how we can solve some of the problems you mentioned, right? Things like latency, right? What if the Wi-Fi drops and things like that? But these, like, bigger models like Astra can still run on the cloud because they don't have to run a couple times per second, right?

[9:20](https://realalpha.app/v/zONxIHfiY9k/?t=560) They can run every second or even slower than that. So we think that an open source model can allow people to optimize that and then install it and run it locally on-device, right? So we think that on-device inference is also going to be very important going forward. So explain it to me, for people who aren't familiar with the robotic space, the hardware space, how does on-device inference work? Why is that super important for robotics?

[9:45](https://realalpha.app/v/zONxIHfiY9k/?t=585) Yeah, 100%. So these modern-day humanoids that you're seeing have, you know, almost half a dozen cameras, whether it's mounted on their head, on their, you know, end effectors, on their grippers, right? And these images get streamed to this really massive model, oftentimes almost 10 billion, you know, parameters, right? That's living on the GPU on the robot. And this model has to run almost as fast as the human reaction time, right?

[10:10](https://realalpha.app/v/zONxIHfiY9k/?t=610) So fun fact, if you play baseball, right, you know the human reaction time is about 200 milliseconds, right? So that means that this model has to take all of these images and produce actions for the robot five times per second, right? So that is very fast, and internet latency sometimes can produce a lot of jitter, right? So when it's a safety-critical system, right, we believe that running it locally can have a lot of benefits, right?

[10:33](https://realalpha.app/v/zONxIHfiY9k/?t=633) So this is kind of a delicate balance between doing that locally plus the added intelligence of a GPT-Astra, right? Is that the industry standard right now where most robots are being run locally? We do see a lot of that today, right? So I was actually just at the conference on robot learning in Austin, Texas, right? When we saw a lot of the demos there, I would say probably half of the company were running, you know, policies locally, right?

[10:59](https://realalpha.app/v/zONxIHfiY9k/?t=659) Whether it's at a desktop, in a booth, right, or just directly on the robot. And when they're running these locally, what models are they typically running? Yeah, good question. So a lot of companies like DeepMind, obviously, right, they have their own leading models. A lot of startups are fine-tuning Pi 0.5, right? So Pi's frontier open source model. But now we are reaching out to more and more partners, especially startups, right, to see if they want to try V0.5 as an alternative, right?

[11:28](https://realalpha.app/v/zONxIHfiY9k/?t=688) It's a fantastic world action model. And we have seen great results fine-tuning that and serving that for different tasks. What are some of those results? Yeah, so we are particularly proud of full-body tasks that we've done, right? So imagine like if you have a robot that is supposed to stock shelves in a grocery store, right? Like you have to bend down and pick up some inventory, put it on the shelf and then turn around, right?

[11:52](https://realalpha.app/v/zONxIHfiY9k/?t=712) So those type of full body tasks we are quite excited about. And I think that also illustrates an area where whether it's. PAI GPT, Astro, or PAI 0.5 are not as particularly good at, right? Who are the current leaders when it comes to local models that are being used in robotics? So we think that NVIDIA Groot is a really fantastic model. They can run fast locally on device. Props to physical intelligence, right?

[12:21](https://realalpha.app/v/zONxIHfiY9k/?t=741) PAI's model, especially their last year's open source model, is also quite strong, right? And I think those are, you know, the two top of mind for us, right? And a lot of the other ones we are just not as aware of, you know, because they're not open source or they're very private or stealthy. But where are they? So compare that to the capabilities of GPT-6. What's the gap there? So on some of these benchmarks, right?

[12:46](https://realalpha.app/v/zONxIHfiY9k/?t=766) So that's a really good question. I think GPT-6 definitely excels at more general tasks, right? So if you are given the robot some previously unseen environment, ChatGPT can do better. But if you are, let's say, strictly benchmarking for cycle time, speed, or other metrics like that, then these special purpose-made robotics models can still win. But we don't think that that's necessarily going to be the long-term future.

[13:12](https://realalpha.app/v/zONxIHfiY9k/?t=792) We think that ChatGPT will also be able to absorb some of those tasks as well. So do you think the future is a mix, a hybrid? So humanoids, I'm just using humanoids as a robot example here, will have a local model that can respond instantly no matter what, even with no Wi-Fi. But then for more difficult tasks and, say, it assesses its situation and says, okay, it's fine if I have a little bit of a lag time, then it goes to a more advanced model in the cloud.

[13:40](https://realalpha.app/v/zONxIHfiY9k/?t=820) Yeah, 100%. I think your intuition there is really spot on. And I think more robotic companies are starting to adopt that formulation here.
