Waymo's reliability comes from decomposed engineering

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 15:51

   talking about so number one i think self-driving is super duper cool um and also i see self-driving as a ginormous engineering win not necessarily an ai win um i use waymo as like the gold standard here no offense to tesla fans but like they seem to be safer and also the way they do it is by building like a la software and engineering like reliable systems that they understand the pieces of that have their own like decomposed abstracted parts and make sure those are like unbelievably good while programming in the behavior of the policies you want such that this can generalize outside of distribution so this is super freaking cool but none of this is happening in ai right now when in ai what you tend to get is a lot more let's just make a demo and you know the the my cynical loop here is make a demo raise like a seed or series a say that you're going to make it

2. 16:49

   reliable never end up making that reliable pivot into a human in the loop version of this thing instead of actually automating the task and and it's not i don't hate on them to be clear that is actually the right move to do given the current ai climate because though models are not reliable enough so even you know i try to be as unbiased as i possibly can i obviously can't be perfect But I would like to have like the same type of critical lens to LLM demos as JEV demos.

3. 17:19

   And I, you know, like I think that a lot of them are super cool, especially when they have sick ideas. And also, I can't really vouch for them because they solve work when they actually reliably can be run in the background. And LLMs are extremely not there. I actually don't know of anyone who like actually runs like an LLM as a separate dependency because you can't get abstraction. with an LM, right?

4. 17:44

   It can just break arbitrarily and you need to have a, the abstraction needs to leak in order to see like, oh, why did you talk about biology? Oh, I'm sorry, my clod can't talk about biology. It fell back to another model or something like that. And that is extremely sensible for a first party app, but super nonsensical in my opinion for something developers are meant to build on top of. So I think that there's that, that lack of usefulness I still think is present there despite demos looking like sci-fi.

5. 18:13

   because demos will always be significantly ahead of the curve. Opening I have seen has been showing like customer service demos since 2020. I believe that is now roughly six years ago and customer service is still not solved. And, you know, maybe because the work is hard, as you say, but it really looks a lot easier than Millennium Prize problems in math because I don't know anyone who could solve a Millennium Prize problem in math.

6. 18:41

   I feel like I know everyone, and I don't know everyone who could solve customer service, but everyone I know probably could do that job better than an LLM can. Or even drive-thrus. People keep trying and failing to automate drive-thrus, and there's no good answer for that other than maybe drive-thrus are harder than unsolved math. It's interesting to me that you call out Waymo versus Tesla in part because In the context of this conversation about like bitter, bitter lesson pilled, right? Because in one sense, like what differentiates Waymo is that they're not taking a bitter lesson pilled, you know, approach to it. Whereas Tesla's is more bitter lesson pilled. Like we're going to, you know, I'm characterizing their approach at a 30,000 foot view, but like we're going to collect a lot of data and throw that into a big model and let it do end to end. Whereas Tesla's, we're going to like classically engineer this thing, break this thing into

7. 19:39

   a lot of different modules, use all the sensors we can to get as much data as we can. I've had conversations with Drago on the podcast about like how they approach it and how it's so fundamentally not like, you know, we're aiming for this end-to-end single model to rule the entire driving experience. So I want to emphasize that that is just the ML part of it, but I don't think that's the limit of what we should be building. And I'm actually extremely pro software in software engineering. And I actually think that I hope that and I want to do everything I can that Jeb will usher in another golden era of software engineering, kind of like with the early internet creative energy of like building crazy things. And for ML, you need data, you need to focus on the right task, you need compute, you need algorithms. But ML is meant to be one part

8. 20:28

   of a larger system, in my opinion, in general. And ML should be scoped to the part of the system that it can really do well. And in general, the way to get higher reliability is to zoom in, you know? Like, would Waymo be as successful in San Francisco if they tried to do the whole thing end-to-end? Honestly, I would wager not. Like, maybe this is going to be better in five years, but in order to, like, actually ship something now that is useful and reliable and safe, I think you need engineering to study all of these things, study the properties of every single ML component, And when things break, be able to like debug which part caused it and fix that.

9. 21:08

   And that's very hard to do in a one model rules them all type setting.