Break AI work into specialized models and tasks

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 59:26

   do you think teams will be building with these researchers on their teams yeah i mean so when you're fine-tuning a model one of the you're basically giving the model a bunch of of examples of the kinds of things you want it to be better at so it's it's here's a problem here's a good answer here's a problem here's a good answer uh or here's a question here's a good answer you know times a thousand or or ten thousand uh and suddenly you're you're teaching the model to be much better than than it was out of the gate at that particular thing we use it everywhere internally um we also we We use ensembles of models much more internally than people might think.

2. 1:00:04

   So it's not, here is, I have 10 different problems. I'll just ask, you know, baseline GPT-4.0 about a bunch of these things. If we have 10 different problems, we might solve them using, you know, 20 different model calls. Some of which are using specialized fine-tuned models. They're using models of different sizes because maybe you have different latency requirements or cost requirements for different questions.

3. 1:00:32

   They are probably using custom prompts for each one. Basically, you want to teach the model to be really good. You want to break the problem down into more specific tasks versus some broader set of high-level tasks. And then you can use models very specifically to get very good at each individual thing. And then you have an ensemble that sort of tackles the whole thing. I think a lot of good companies are doing that today.

4. 1:00:57

   I still see a lot of companies kind of giving the model single generic broad problems versus breaking the problem down. And I think there will be more breaking the problem down using specific models for specific things, including fine tuning. And so in your case, because this is really interesting, is that you're using different levels of ChatGPT, like 01, 03 and stuff that's earlier. there'll be parts of our internal stack so we do if you give you an example uh customer support with 400 plus weekly uh 400 plus million weekly active users we get you know a lot of inbound tickets right i don't know how many customer support folks we have but it's not very many 30 40 i'm not sure way way smaller than you would have at any comparable company and it's because we've automated a lot of our flows, we've got, you know, most questions, using our internal

5. 1:01:56

   resources, knowledge base, you know, guidelines for how we answer questions, what kind of personality, etc. You can teach the model those things, and then have it do a lot of its answers automatically, or where it doesn't have, you know, the full confidence to answer a particular question, it can still suggest an answer, request a human to look at it, and then that human's answer actually is its own sort of fine tuning data for for the model you're telling it the right answer in a particular case and uh and we're using it various places you know some of these places you want a little bit more reasoning is not super latency sensitive so you want a little more reasoning and we'll use one of our o-series models in other places you want a quick check on something and so you're fine to use like 40 mini which is super fast and super cheap and in general

6. 1:02:46

   it's like specific models for specific purposes and then you you you ensemble them together to solve problems by the way again not unlike how we as humans solve problems a company is arguably an ensemble of models that have all been you know fine-tuned and based on what we studied in college and what we have like learned over the course of our careers we've all been fine-tuned to have different sets of skills and you like group them together in different configurations and the output of the ensemble is much better than the output of any one individual kevin you're blowing my mind that sounds exactly correct uh and also different people are you pay them less uh they cost less to talk to some people take a long time to answer some people hallucinate this is this is like uh this is a mental model that really does work in in thinking nothing this is great

7. 1:03:42

   Right. Some people are visual. They want to draw out their thinking. Some people want to talk word cell. Wow. This is a really good metaphor. So again, coming back to your advice here, because I love that we circled back to it. It's you're finding a really good way to think about how to design great AI experiences and LMs, I guess, specifically to think about how a person Do this? Well, it's maybe not always the answer is to think about how a person would do it, but sometimes to gain intuition for how you might solve a problem, you think about what an equivalent human would do in those situations and use that to at least gain a different perspective on the problem. Wow, this is great.

8. 1:04:20

   There's just like, because so much of this really is talking to a model. There's a lot of prior art