A stable scorecard can show a system is nearing deployment

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 37:34

   before you even thought of like building a product the demo only took about 18 months but the actual product took about 15 years so for everyone in this room where the demo actually works what are the signals you need in like you know maybe a year or two that tell you you're on a 15 year problem solving trajectory or like you know you're working on a system that like you could produce a workable product in the next three years and i get the fully autonomous versus not is a good like rubric but are there other things let's say you're working on a problem uh you said you know three or 15 years let's say you know you've been working on it maybe you know for a year or two and you kind of want to see if you're a year away or 10 plus years away.

2. 38:20

   One thing, kind of one pattern that I've seen over and over again is do you have convergence of discovery? Is the problems that you are seeing today, how different are they from the problems you saw six months ago? Do you feel like as you're pushing on the system, does it feel like you have stability and you're just kind of iteratively improving your metrics in the same areas or does it feel like every six months the landscape is shifting from under you and the problems of today are very different from what you saw a while ago?

3. 38:55

   So if you don't have that stability, then that's often kind of a good indicator that you've got some ways to go. The second thing I would mention is judging how close you are to a real deployment, a real product is, again, I go back to metrics and evaluation, and do you have a scorecard? If you think you're six months or a year away from deployment, do you have a scorecard that if it were green today would allow you to deploy with confidence?

4. 39:28

   If you don't have one, you're saving that for later. Often that turns out to be more difficult than you expected and there's a long distance to go. But if you already have one and you're seeing stability in your discovery and stability in your metrics and it's just a matter of closing the gap, that is a really good indicator that you are getting pretty close. Those are both really good, especially tying that back to what you said in the beginning, which was when you started at Waymo, you weren't trying to build a product.

5. 39:58

   You were just like, we were trying to figure out feasibility and we set out some very concrete milestones and goals. And I'm assuming you must have seen that you were able to make progress without it kind of shifting from underneath you to kind of give you confidence to keep on laddering up from there. We were not at that stage. We weren't at that. And as I said, this was about these goals that we picked.

6. 40:20

   This was not in service of building a product. This was in service of us learning. Just learning, okay. And I guess this is where I would make a clear distinction, right? In some ways, if you take my own advice, you're working a hard problem, you start with the fundamentals before you start specializing and thinking about the product or an application. That was not the very first thing we did. But that thing was not in service of, you know, that integration of all the different components at the very, very early stage was not because we thought we were going to be that close and we were going to tweak it and have a product in six months.

7. 40:51

   It was because we needed to bring everything together to really deeply understand the complexity of the task ahead of us.