Superposition lets networks represent more features than dimensions

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 4:40:16

   the world. So another interesting hypothesis is the superposition hypothesis. Can you describe what superposition is? Yeah. So earlier we were talking about word to fact, right? And we were talking about how maybe you have one direction that corresponds to gender and maybe another that corresponds to royalty and another one that corresponds to Italy and another one that corresponds to, you know, food and all of these things.

2. 4:40:35

   Well, you know, oftentimes maybe these word embeddings, they might be 500 dimensions, 1,000 dimensions. And so if you believe that all of those directions were orthogonal, then you could only have, you know, 500 concepts. And, you know, I love pizza. But like, if I was going to go and like give the... like 500 most important concepts in, you know, the English language, probably Italy wouldn't be, it's not obvious at least that Italy would be one of them, right? Because you have to have things like plural and singular and verb and noun and adjective. And, you know, there's a lot of things we have to get to before we get to Italy and Japan. And, you know, there's a lot of countries in the world. And so how might it be that models could, you know, simultaneously have the linear representation hypothesis be true, and also represent more things than they have directions?

3. 4:41:29

   So what does that mean? Well, okay, so if linear representation hypothesis is true, something interesting has to be going on. Now, I'll tell you one more interesting thing before we go and we do that, which is, you know, earlier we were talking about all these polysomatic neurons, right? And these neurons that, you know, when we were looking at Inception V1, there's these nice neurons that like the car detector and the curve detector and so on that respond to lots of, you know, to very coherent things. But it's lots of neurons that respond to a bunch of unrelated things. And that's also an interesting phenomenon. And it turns out as well that even these neurons that are really, really clean, if you look at the weak activations, right? So if you look at like, you know, the activations where it's like activating 5% of the, you know, of the maximum

4. 4:42:10

   activation, it's really not the core thing that it's expecting, right? So if you look at a curve detector, for instance, and you look at the places where it's 5% active, you know, you could interpret it just as noise, or it could be that it's doing something else there. Okay, so how could that be? Well, there's this amazing thing in mathematics called compressed sensing. And it's actually this very surprising fact, where if you have a high dimensional space, and you project into a low-dimensional space, ordinarily, you can't go and sort of unproject it and get back your high-dimensional vector, right? You threw information away. This is like, you know, you can't invert a rectangular matrix. You can only invert square matrices. But it turns out that that's actually not quite true. If I tell you that the high-dimensional vector was sparse,

5. 4:43:00

   so it's mostly zeros, then it turns out that you can often go and find back the high-dimensional vector with very high probability. So that's a surprising fact, right? It says that, you know, you can have this high-dimensional vector space, and as long as things are sparse, you can project it down, you can have a lower-dimensional projection of it, and that works. So the series' hypothesis is saying that that's what's going on in neural networks. For instance, that's what's going on in word embeddings. The word embeddings are able to simultaneously have directions be the meaningful thing, and by exploiting the fact that they're they're operating on a fairly high dimensional space. They're actually, and the fact that these concepts are sparse, right? Like, you know, you usually aren't talking about Japan and Italy at

6. 4:43:44

   the same time. You know, most of the, most of those concepts, you know, in most instances, Japan and Italy are both zero. They're not present at all. And if that's true, then you can go and have it be the case that, that you can, you can have many more of these sort of directions that are meaningful, these features than you have dimensions. And similarly, when we're talking about neurons, you can have many more concepts than you have have neurons. So that's at a high level of superstition hypothesis. Now, it has this even wilder implication, which is to go and say that neural networks are, it may not just be the case that the representations are like this, but the computation may also be like this, you know, the connections between all of them. And so in some sense, neural networks may be shadows of much larger, sparser neural networks. And what

7. 4:44:34

   we see are these projections. And the strongest version of the superstition hypothesis would be to take that really seriously and sort of say, you know, there actually is in some sense this upstairs model, you know, where the neurons are really sparse and all-interpol and there's, you know, the weights between them are these really sparse circuits. And that's... That's what we're studying. And the thing that we're observing is the shadow of it.

8. 4:45:00

   And so we need to find the original object. And the process of learning is trying to construct a compression of the upstairs model that doesn't lose too much information in the projection. Yeah, it's finding how to fit it efficiently or something like this. The gradient descent is doing this. So this sort of says that gradient descent, it could just represent a dense neural network,