Closed-loop validation requires a driver, simulator and critic

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 15:06

   how would you explain the Waymo end-to-end system? So there's a couple of things. End-to-end has a specific meaning in the field, so I can speak to that. The way I would describe the, of course, on the car, it's a system that goes from sensors. We use three main sensing modalities, cameras, lighters, and radars. That information gets processed and kind of encoded into a more compressed representation. And then there's a generative part, which on the car is the part that makes decisions and generates driving behaviors. That's the decoder or the planning part. All of that, the thing that's running in the car that we call the Waymo driver, that's just one part of the system. Really, when we think about the technical challenge that we're solving, there's three main pillars. We call them the driver, the simulator, and the critic.

2. 16:05

   And it starts with us with building large, high-capacity off-board models. We have three teachers for each one of those tasks. And then they get distilled into their students. And the The driver acts, the simulator generates features, often in a synthetic environment, and the critic judges and evaluates. All of them have to work together in order for this ecosystem to move forward. Do they all run on the car itself?

3. 16:39

   Does the simulator need to run on the car or can it run on the cloud? Oh, no, absolutely not. The only thing that runs on the car is the driver. and the other two parts are in service of training, evaluating, and validating the driver. Do you think that the architecture that you have described is an emergent one based on how you have seen almost the path dependence of the learnings over the last 17 years of building this?

4. 17:06

   Or would you almost derive this kind of three models, kind of like the simulator, the driver, the critic, does that feel like the right shape of architecture for real-world, low-latency, high-feedback loops for a variety of different kind of devices and so on? Yeah, I think so, with some caveats. And I think the key distinction and the key aspect of the problem, whether you need all three, will be the level of complexity of the task.

5. 17:44

   And the key distinction will be whether you can get by with kind of open-loop imitation and open-loop evaluation and validation, which for some applications is sufficient, not in our case. And if it's not sufficient, then you do need the entire ecosystem. So the way in our space, I think you'll see the same property emerge in other fully autonomous applications of AI in the physical world where you cannot sufficiently control the environment or make the problem simpler.

6. 18:19

   If open-loop imitation and open-loop evaluation and validation is insufficient, that means that closed-loop is required for validation, as in the case of our product. And it is incredibly valuable for training. There are some things you can do in closed-loop training that you cannot do in open-loop. Then that leads you to needing a high-fidelity realistic simulator. Number one. And number two, if you are playing in this closed-loop training ecosystem, then the separation between the actor and a critic, as often happens in reinforcement learning frameworks, is a very natural one.

7. 18:59

   So then if you are in this closed-loop as required, then I think you very naturally end up with the trifecta. You've often said that the hard part of driving isn't just the perception.