Models form concepts and plan beyond the next word

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 11:49

   I'm curious what else, as you spend time paying attention to how LLMs work and operate, I feel like you're in the top 0.1% of people that have studied and watched LMs operate and think. What else has surprised you about how they operate and think? there's so much um so maybe i'll go through like rather than you know there's there's a bunch of findings but then maybe there's sort of like what yeah how they've changed how i think about models so to go back to the to the you know capital example uh this sort of like you know ask the model you know the capital of the state that contains dallas is so one question that that people have had for a while is how do these models answer questions like this is it that they've seen just so much text that they've kind of memorized like lists of you know prompts and responses

2. 12:43

   or is it that they're having some sort of model of of like what you're actually asking about that you're actually asking back like cities and cities are in states and states are in countries and things like that um and so one thing that that we do on my team is is we ask a question like this And then we look inside the model at these intermediate results. Before it produces a given word, you know, as it's about to answer, we just look at the intermediate results, and we basically have an immense amount of effort dedicated just trying to decode what these intermediate results mean.

3. 13:18

   And if we decode them well, which we can do in some cases, we can see that models often have and represent these concepts. So when you ask that Dallas question, what you see is inside the model, you see the representation, you see an intermediate result that maps the representation of Texas that lights up and one to capitals. And then those two combine to activate an intermediate result. It's like, oh, the capital of Texas, that's Austin.

4. 13:45

   They have representations like this for basically everything, like every concept that you can think of, or at least the vast majority of the ones we've tried. they sort of represent you know these concepts uh uniquely across contexts so one thing i find really neat is um there's this there's this debate about you know our language models like thinking and then you know that's that's a hard question but one aspect of thought is like having unique representations that are shared when you think about different things so you know uh you you know what the concept of an eye is.

5. 14:22

   And it could be like an eye in a photo or an eye in like a medical diagram or an eye in a little drawing. And we see that in language models as well. If you pass a prompt that has a photo of an eye or you talk about eyes in any language or you draw a little SVG with an eye in it and you look at the intermediate results inside the model, the same direction lights up and that direction is like, oh, this is an eye.

6. 14:43

   And so there's a sort of a sense in which they have these general concepts that they know about and that they use and reuse to solve problems. Another thing that sort of is surprising to people, it's kind of the mirror of what I was saying about, you know, I'm saying the sentence saying one word at a time, but I have some destination in mind. So one mystery of language models is, man, if they're outputting only one word at a time, how is it even possible that they could, you know, write a coherent report or like write code well?

7. 15:16

   It's sort of like no one on earth that is doing this as their job, you know, that's like writing reports or, you know, writing emails is literally just only thinking about the next word in the sentence with no idea, you know, like empty mind about anything that comes next. And so that was sort of a big question that we had for a while. And if you look inside these models, one neat fact about these models is that all of the intermediate results that they've produced when they were predicting a word, let's say the fifth word that they predicted, they can be reused.

8. 15:50

   It can be accessed and reused by all of the future words. So what does that mean? That means that in practice, we find that models will, in their intermediate results, sometimes formulate a plan. The example we have for this is poetry. It's hard to rhyme without planning. You kind of have to think of like, oh, I just ended my first rhyme in like an it word. And so I should end my second line with another it word.

9. 16:15

   You know, maybe the word I should end it with is rabbit. And we find that if you look at models as they're like writing poetry, you see exactly that. They write a line and then like in their intermediate results, they're like, oh, oh, oh, this is an it rhyme. I should rhyme with rabbit. And then every future word that they predict looks back to that plan they made. And so even though they're outputting one word at a time, just like I am right now, they can, of course, have a plan that they follow over many, many, many, many words.

10. 16:40

    And so all of a sudden, even though you have a next word predictor, even though it's technically right, you actually have something that's closer to being able to just do higher order tasks where it's like, oh, well, you just have a plan and you just kind of refer back to your plan and you're slowly executing on it. And you can do that without saying the plan. That's the cool part, is like this plan that we find is in these intermediate results inside the model, but the model doesn't even have to sort of like say it out loud.

11. 17:07

    Wow.