Reliability matters more than inventing new model shapes

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 18:04

   I think new shapes are freaking awesome we plan to keep shipping them I'm a little bit worried that again what we talked about maybe this was like before we got on the record that ML people especially have like kind of like lost the actual message i was trying to say um and i'm worried that people are now over focused on like the shapes of tasks rather than the reliability the shape of task is incredibly you know like it's super easy to do that when i thought this would take a week i thought it would take a week to make the shape of the task which it did and this is what's happening now with other people making similar models with similar shapes but the shape really doesn't matter compared to actually it being useful and that is the thing i think people really really need to focus on so there will 100 be way more shapes than these two

2. 18:51

   even if no one else does it we will do them i think people are making too many new shapes right now and not caring enough about you like the utility of it and i'm worried that this will actually burn the goodwill that people people don't like being uh over promised under delivered to and like these things are like pure over promised under deliver and i think that that's like not good for the world you know like if people have like creative playful energy we don't want them to waste it on stuff that clearly doesn't work. As far as I can tell, people dislike that. And even if they dislike it, I hate it. Like I, oh man. And I think there's going to be a balance. There will definitely be a lot more future shapes to be had. And I just really, really hope that people start giving like unbelievable levels of fucks for

3. 19:38

   utility, usefulness, reliability. What would a developer want in this situation rather than ml people who i think focus on like i want to flag plant you know like i want to be the one who names the thing that gets cited i want to like get um revenue in the short term because people will play around with it and i want to do the same thing that every other ai company does I really hope we don't do this, but overpromising.

4. 20:00

   under deliver in order to get attention because it feels like a lot of the ai economy ish is really driven by that attention and that is so so bad for developers it's like actually the absolute worst for the developers who do not have like the time to test these things themselves because they just get they're the ones who get swept up in all this and this is also why i'm anti-benchmarks it's a it's a it's so bad for those people you know you have to play around with some of this stuff yourself to have any sense of of how good it actually is yes and like that is tragic you know like it makes sense um like i actually think that as far as i can tell people have stopped caring about like benchmarks for our class of model maybe i'm like in the wrong um uh niche for this but my sense and there could be survivorship bias is that people already don't care about new releases here

5. 20:52

   because they've all been lied to so many times. And this is good for us as a company, but bad for the field because actually if someone makes a genuinely good thing, there should be a way to acknowledge it and shout it out and say like, hey, this is awesome. But like in the world where everyone optimizes for benchmarks and demos and things that don't require liability, it's very hard to communicate that other than this very manual process of evaluating models themselves, which is actually super duper hard because benchmarks also don't capture that.

6. 21:23

   One thing I'm struck by, you know, is obviously this idea of like these shapes of problems, right? And the chatbots are very good for some shape of a problem you're trying to solve.