Each additional safety nine demands more rigorous evaluation

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 27:06

   Performance and reliability. It's getting all those nines in the safety of your system, and reliability is one of the key important parts of it. So I've heard, but not experienced, that every nine takes 10x the effort. That tends to be roughly the rule, yeah. So which nine was the hardest? I mean, for those who don't know what a nine is, can you? If you look at whether it's the performance of the system, pick anything around safety or quality.

2. 27:39

   If you look at the reliability of the system, you have some metrics, and you want to get to a certain number of nines for it to be deployable with confidence in the physical world. and again, this is another way to describe that property that I mentioned where, you know, kind of start out steep, and it's very easy to get to the first 90%, and then it's, you know, that much more difficult to get to the next, you know, 99, 99.9999, right?

3. 28:05

   And the way you end up solving those subsequent nines oftentimes requires, you know, very different approaches than the previous ones. So your question was, which one is the hardest? Well, okay, the one you're working on right now is always the kind of the hardest, but also the most interesting, the most exciting one. But maybe that's... Which nine let you deploy Waymos at such a large scale? Well, I guess the first big step was to...

4. 28:35

   We kind of had our usual zero to one moment and then one to 10, 100, and so forth. The zero to one, that was a really big step. That was in 2015 when we, for the first time, operate a vehicle without a human driver on public roads. So that was a huge step. This was, and this is kind of this massive transition from a demo or a driver assist product to a full autonomous system. So that was definitely a step function.

5. 29:09

   That step forced us to think very deeply about the performance and reliability of the system across the whole stack from the platform itself, the car, you know, how reliable is the car? Do you have redundant actuation? You know, what if the brakes fail? What if the steering fails? What if you lose power? The hardware that we're putting on the car, the sensors on the compute, you know, do we have redundancy there? Do we have the right number of nines in that part of the stack as well as the software that we are putting on the vehicle? And very importantly, at that time, we had to, you know, deeply think about this question of what does it mean for an autonomous vehicles to be ready. What's good enough?

6. 29:50

   So this was the first version of what we refer to as our safety and readiness framework, which is this robust, multifaceted machine that we've built over the years that is data-driven that allows us to repeatedly ask this question of, is the system ready and can we deploy? So that was 2015. I think most of us who have a more distributed systems back end or kind of a world where we have kind of built apps for a living.

7. 30:22

   We're used to living with a world where failures happen. You tolerate them. It's not that big a deal. You kind of recover them. Most of the systems are pretty fault-tolerant slash the worst case is not that big a deal. So what is a mindset shift you need to make to build systems for the real world, deploying two-ton robots that are flying through San Francisco? Do you have to untrain people who you hire to kind of think differently?

8. 30:46

   how do you put that mindset in? I think it's a very different mindset that most of us are used to when we build computer systems. Generally, you don't have to untrain people. You have to hire people who are very good and are adaptive and fast learners. But you get, I guess, some of the principles that we apply to our research and engineering are not earth-shattering. we just take them very seriously and we apply them to that very high degree of rigor in everything we do.

9. 31:23

   And it starts with having that culture of being safety first and eval data driven primarily. And just like safety eval is not something you do after the fact. It's not something that you do is an afterthought. It is something that permeates all of the work you do across the team and all of your engineering practices. But it's not unlike good engineering practices of being test-driven development. You start with the test before you write your code.

10. 31:58

    You think about what has to be true and how do you evaluate your system before you deploy. We just take it to that next level of fidelity. The part that I'm still wrapping my head around is when a robot learns in simulation, how do you know whether it's getting better or getting better at exploiting a loophole? Like, you know, like what is the reward function? Like what evidence do you require outside of the training group to prove that, you know, to believe that the improvement is actually real?

11. 32:27

    Yeah, so I think this gets back to this, you know, fundamental question of metrics and evaluation. I have to ask right after this, how strong is your eval team? We've hit upon evals so much. Yeah, we have, you know, a very strong eval team. the team that builds the infrastructure, the simulator, all of the metrics, the data science team that supports that. But it's also like the eval-first mindset that is spread across the entire company, across definitely all of the engineering functions.

12. 32:57

    And you do, it's not one or two simple metrics that would describe that, what does it mean for an autonomous system to be ready and good enough? and it's not a couple of things that you measure at the system level at the very end of your development. It starts, you do eval and hard data-driven evaluation every part of your system, right? It starts with that model level. Whatever model you're training and you don't train it monolithically and you train your encoders, you train your decoder and they have many metrics.

13. 33:30

    We can talk a little bit about end-to-end and structured representation because that is an important aspect of supporting rigorous training. and evaluation. And then that goes to dense proxy metrics of the system level performance that are maybe not the final things you care about, but they're very dense predictive signals about the performance that you will see in the field. And then you do rigorous system level evaluation, everything from safety to quality. And we run billions of miles of simulation and every release that before we put it out on the road goes through this multi-week rigorous evaluation and validation cycle that includes testing both in the physical world and in simulation.

14. 34:12

    And then we have a team of well, highly experienced subject matter experts that review all of the data and all of that gets fully vetted before we put the software on railroads. I mean, to your point earlier, it feels as though that having that pipeline and having that infrastructure to be able to both iterate and then deploy with confidence feels like a real source of competitive advantage. And I suspect that anybody who's deploying any kind of physical, whether robots, drones, and so on, at scale will have to have that.

15. 34:48

    But I don't know if there are that many companies that have that today, right? Yeah, I think that's one of the hardest things to build. And really, again, there are some applications where that might not be required.