Build AI products around the model's expanding abilities

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 1:03:22

   You shared some really cool tips for how to get the most out of AI, how to build on AI, how to build great products on AI. One tip you shared is give your team as many tokens as they want, just like let them experiment. You also shared just advice generally of just build towards where the model is going, not to where it is today. What other advice do you have for folks that are trying to build AI products?

2. 1:03:42

   I'd probably share a few more things. So one is don't try to box the model in. I think a lot of people's instinct when they build on the model is they try to make it behave a very particular way. They're like, this is a component of a bigger system. I think some examples of this are people wearing very strict workflows on the model, for example. You know, to say like, you must do step one, then step two, then step three.

3. 1:04:03

   And you have this very fancy orchestrator doing this. But actually, almost always you get better results if you just give the model tools, you give it a goal and you let it figure it out. I think a year ago, you actually needed a lot of the scaffolding, but nowadays you don't really need it. So, you know, I don't know what to call this principle, but it's like, you know, like ask not what the model can do for you.

4. 1:04:21

   Maybe it's something like this. Just think about how do you give the model the tools to do things? Don't try to over curate it. Don't try to put it into a box. Don't try to give it a bunch of context up front. Give it a tool so that it can get the context it needs. You're just going to get better results. I think a second one is maybe actually like a more even more general version of this principle is just the bitter lesson.

5. 1:04:45

   And actually for the Quadco team, hopefully listeners have read this, but Rich Sutton had this blog post maybe 10 years ago called The Bitter Lesson. And it's actually a really simple idea. His idea was that the more general model will always outperform the more specific model. And I think for him, he was talking about like self-driving cars and other domains like this. But actually, there's just so many corollaries to the bitter lesson.

6. 1:05:10

   And for me, the biggest one is just always bet on the more general model. and over the long term, don't try to use tiny models for stuff. Don't try to fine tune. Don't try to do any of this stuff. There's some applications. There's some reasons to do this, but almost always try to bet on the more general model if you can, if you have that flexibility. And so these workflows are essentially a way that it's not a general model. It's putting the scaffolding around it. And in general, what we see is maybe scaffolding can improve performance maybe 10 20 something like this but often these gains just get wiped out with the next model so it's almost better to just wait for the next one and i think maybe this is a final principle and something that cloud code i think got right in hindsight from the very beginning we bet on

7. 1:05:58

   building for the model six months from now not for the model of today and for the very early versions of the product, they just wrote so little of my code because I didn't trust it. Because, you know, it was like Sonnet 3.5, then it was like 3.6 or forget 3.5 new, whatever name we gave it. These models just weren't very good at coding yet. They were getting there, but it was still pretty early. So back then the model used Git for me, it automated some things, but it really wasn't doing a huge amount of my coding. And so the bet with quad code was at some some point the model gets good enough that it can just write a lot of the code. And this is the thing that we first started seeing with Opus 4 and Sonnet 4. And Opus 4 was our first kind of ASL 3 class model that we released back in May. And we just saw this inflection because everyone started to

8. 1:06:49

   use quad code for the first time. And that was kind of when our growth really went exponential. And like I said, it stayed there. So I think this is advice that I actually give to a lot of folks, especially people building startups it's going to be uncomfortable because your product market won't be very good for the first six months but if you build for the model six months out when that model comes out you're just going to hit the ground running and the product is going to click and start to work and when you say build for the model six months out what is what is it that you think people can assume will happen is it just generally it will get better at things is it just like okay it's like almost good enough and that's a sign that it'll probably get better at that thing? Is there any advice there? I think that's a good way to do it. Like,

9. 1:07:32

   you know, obviously within an AI lab, we get to see the specific ways that it gets better. So it's a little unfair, but we also, we try to talk about this. So, you know, like one of the ways that it's going to get better is it's going to get better and better using tools and using computers. This is a bet that I would make. Another one is it's going to get better and better for long periods of time. And this is a place, you know, like there's all sorts of studies about this. But if you just trace the trajectory, or, you know, maybe even like for my own experience, when I used Sonnet 3.5 back, you know, a year ago, it could run for maybe 15 or 30 seconds before it started going off the rails. And you just really had to hold its hand through any kind of complicated task. But nowadays with Opus 4.6, you know, on average,

10. 1:08:16

    it'll run maybe 10, 30, 20, 30 minutes unattended. And I'll just like start another quad and have it do something else. And like I said, always have a bunch of quads running. And they can also run for hours or even days at a time. I think there are some examples where they ran for many weeks. And so I think over time, this is going to become more and more normal, where the models are running for a very, very long period of time. And you don't have to sit there and babysit them anymore.