Waymo’s moat is its system for improving the product

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 22:13

   How do you rewrite a system, especially when it already works? That's a great question. Look, architecture is kind of come and go. I just mentioned a few really big waves of innovation that we've seen. The foundation is kind of all of the machinery that allows you to grab the fruits of the latest state-of-the-art technology at any moment in time. And that's your evaluation, that's your data, that's infrastructure.

2. 22:52

   If you don't have that, if you don't have those strong fundamentals, then you kind of evolve into a tech stack that kind of works and has certain implicit properties that have been organically merged over time. But it can be very, very difficult, if not impossible, to make big changes to it. And there's plenty of companies, I think, in the history of technology development that kind of got stuck in those local minima.

3. 23:21

   So what you want, if you have, conversely, if you have those strong fundamentals, that allows you to take really big technical swings. again, whether it's transformers or ConvNets or what have you, and land big disruptive change in technology without the disruption of momentum of your product and without seeing regressions in the field, which, of course, is so important for us because we're operating in a safety-critical environment.

4. 23:50

   So, if anything, I would say that's probably the most difficult thing that we've built at Waymo and probably the most important muscle that we as a company develop over time It's not, again, the technology of the day. It's this ability to bet on the latest and greatest innovation and then take big disruptive steps in technology development without breaking momentum of product development and deployment.

5. 24:14

   So to repeat, the key would be infrastructure, data, and then evaluation. That's right. Given the model that you have trained, and then you have distilled two kind of these cars, if that was the only IP that Waymo had today, Like, okay, you have these models that have been trained, but you didn't actually have kind of like the historical trove of essentially evals and data and so on. Would that be an asset, like, would that be a sufficient asset to start kind of like a new AV company?

6. 24:43

   I can give you the model that we have right now. I don't think you would know what to do with it. You wouldn't be able to validate it. If you wanted to go and apply it to a new deployment, I don't think you would have the machinery to validate it and deploy with confidence, let alone iterate on it. So it really comes down to the fundamentals of infrastructure, metrics, eval, training recipes. It's not static.

7. 25:07

   It's all about how quickly you can iterate and your level of confidence and understanding the properties of the thing you're building. So it's much more about the machine for building the product as opposed to a snapshot of the product you have at any moment in time. And this is an industry where it's always been very easy to get started. In 2009, it was a dozen of us, in a course of about 12 to 18 months, built a system that could drive those 10 routes that were very difficult and with no human invention done each.

8. 25:49

   But it's been 17 years to get to where we are today. and it's kind of the natural property of many of these technical curves, right? They start out steep and then they flatten out. And with every breakthrough that we've seen over the years in technology, whether it's hardware, computers, off-board, on-board, ML, massive, it really has a drastic impact on the early part of the curve and getting started has, I think, been getting easier, right?

9. 26:16

   So nowadays you can take your favorite model, a state-of-the-art, some VLM or VLA. Ask me. Yeah, there you go. Pick your favorite one and go from kind of do open loop end-to-end imitation learning on it. And you will very quickly with a tiny budget and a tiny team in a matter of weeks, if not days, get something that works. You mean something that works and drives? It's not going to be good enough for a product, but it will be very impressive.

10. 26:49

    It has a drastic impact on that early part of the curve. but it doesn't make, if you're talking about superhuman performance, if you're talking about deployment at scale, it has a much muted impact on the hard part of the problem. And is the hard part reliability?