Synthetic training data can narrow a model's range of thought

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 51:57

   why it's not trivial. So if I can just describe one, why can't we just synthetically generate and train on it? Well, because every synthetic example, like if I just give synthetic generation of the model thinking about a book, you look at it and you're like, this looks great. Why can't I train on it? Well, you could try, but the model will actually get much worse if you continue trying. And that's because all of the samples you get from models are silently collapsed.

2. 52:18

   They're silently, this is not obvious if you look at any individual example of it, they occupy a very tiny manifold of the possible space of sort of thoughts about content. So the LLMs, when they come off, they're what we call collapsed. They have a collapsed data distribution. If you sample, one easy way to say it is go to chat GPT and ask it, tell me a joke. It only has like three jokes. It's not giving you the whole breadth of possible jokes.

3. 52:41

   It's giving you like, it knows like three jokes. They're silently collapsed. So basically, you're not getting the richness and diversity. and the entropy from these models as you would get from humans. So humans are a lot more sort of noisier, but at least they're not biased. They're not in a statistical sense. They're not silently collapsed. They maintain a huge amount of entropy. So how do you get synthetic data generation to work despite the collapse and while maintaining the entropy is a research problem?

4. 53:07

   Just to make sure I understood, the reason that the collapse is relevant to synthetic data generation is because you want to be able to come up with synthetic problems or reflections which are not already in your data distribution? I guess what I'm saying is, say we have a chapter of a book and I ask an alum to think about it. It will give you something that looks very reasonable. But if I ask it 10 times, you'll notice that all of them are the same.

5. 53:31

   You can't just leave scaling, quote unquote, reflection on the same amount of prompt information and then get returns from that. Yeah, yeah, yeah. So any individual sample will look okay, but the distribution of it is quite terrible and it's quite terrible in such a way that if you continue training on too much of your own stuff you actually collapse I actually think that there's no like fundamental solutions to this possibly and I also think humans collapse over time I think this is again these analogies are surprisingly good but humans collapse during the course of their lives this is why children have completely you know they haven't overfit yet and they will say stuff that will shock you because it's kind of you can see where they're coming from but it's just not the thing people say and because they're

6. 54:12

   not yet collapsed But we're collapsed. We end up revisiting the same thoughts. We end up, you know, saying more and more of the same stuff. And the learning rates go down and the collapse continues to get worse. And then everything deteriorates. Have you seen a super interesting paper that dreaming is a way of preventing this kind of overfitting and collapse? That the reason dreaming is evolutionary adaptive is to put you in weird situations that are very unlike your day-to-day reality to prevent this kind of overfitting?

7. 54:44

   It's an interesting idea. I mean, I do think that when you're generating things in your head and then you're attending to it, you're kind of like training on your own samples. You're training on your synthetic data. And if you do it for too long, you go off rails and you collapse way too much. so you always have to like seek um entropy in your life yeah uh so talking to other people is a great source of entropy and uh things like that so maybe the brain has also built some internal mechanisms for increasing the amount of entropy um in in that process but yeah maybe that's an interesting idea this is a very ill-formed thought so i'll just put it out and let you react to it the best learners that we are aware of which are children are extremely bad at recollecting information.

8. 55:25

   In fact, at the very earliest stages of childhood, you will forget everything. You're just an amnesiac about everything that happens before a certain year date. But you're like extremely good at picking up new languages and learning from the world. And maybe there's some element of like being able to see the forest for the trees. Whereas if you compare it to the opposite end of the spectrum, you have LLM pre-training, which these models will literally be able to regurgitate word for word what is the next thing in a Wikipedia page.

9. 55:49

   But their ability to learn abstract concepts really quickly the way a child can is much more limited. And then adults are somewhere in between where they don't have the flexibility of childhood learning. But they can, you know, adults can memorize facts and information in a way that is harder for kids. And I don't know if there's something interesting about that. I think there's something very interesting about that.

10. 56:09

    Yeah, 100%. I do think that humans actually, they do kind of like have a lot more of an element compared to LLMs of like seeing the forest for the trees. And we're not actually that good at memorization. which is actually a feature. Because we're not that good at memorization, we actually are kind of like forced to find the patterns like in a marginal sense. I think LLens in comparison are extremely good at memorization.

11. 56:34

    They will recite passages from all these training sources. You can give them completely nonsensical data. Like you can take, you can hash some amount of text or something like that. You get a completely random sequence. If you train on it, even just I think a single iteration or two, it can suddenly regurgitate. entire thing he'll memorize it there's no way a person can read a single sequence of random numbers and recite it to you um and that's a feature not a bug almost because it forces you to like only learn the generalizable components whereas lms are distracted by all the memory that they have of the preaching documents and it's probably very distracting to them uh in a certain sense so that's why when i talk about the cognitive core i actually want to remove the memory which is what we talked

12. 57:12

    about i'd love to have them have less memory so that they have to look things up uh and that they only maintain the algorithms for like thought and the idea of an experiment and all this cognitive glue of of acting and this is also relevant to preventing model collapse um let me think um i'm not sure i think it's almost like a separate axis it's almost like the models are way too good at uh memorization and somehow we should we should remove that and i think people people People are much worse, but it's a good thing.

13. 57:45

    What is a solution to model collapse? I mean, there's very naive things you could attempt. It's just like the distribution over loges should be wider or something. Like there's many naive things you could try. What ends up being the problem with the naive approaches? Yeah, I think that's a great question. I mean, you can imagine having a regularization for entropy and things like that. I guess they just don't work as well empirically because right now, like the models are collapsed.

14. 58:09

    But I will say most of the tasks that we want of them don't actually demand the diversity. It's probably the answer of what's going on. And so it's just that the frontier labs are trying to make the models useful. And I kind of just feel like the diversity of the outputs is not so much, number one, it's much harder to work with and evaluate and all this kind of stuff. But maybe it's not what's actually capturing most of the value.

15. 58:31

    In fact, it's actively penalized, right? If you're like super creative in RL, it's like not good. Yeah. Or like maybe if you're doing a lot of writing help from LLMs and stuff like that, I think it's probably bad because the models will give you these like silently, all the same stuff, you know. So they're not, they won't explore lots of different ways of answering a question, right? But I kind of feel like maybe this diversity is just not as big of a, yeah, maybe like, yeah, not as many applications needed so the models don't have it, but then it's actually a problem.

16. 58:57

    It's in third generation time, et cetera. So we're actually shooting ourselves in the foot by not allowing this entropy to maintain in the model. And I think possibly the labs should try harder. And then I think you hinted that it's a very fundamental problem. It won't be easy to solve. What's your intuition for that? I don't actually know if it's super fundamental. I don't actually know if I intended to say that.

17. 59:18

    I do think that, I haven't done these experiments, but I do think that you could probably regularize the entropy to be higher. So you're encouraging the model to give you more and more solutions. But you don't want it to start deviating too much from the training data. It's going to start making up its own language. It's going to start using words that are extremely rare. you know so it's going to drift too much from the distribution so i think controlling the distribution is just like a tricky it's just like someone just has to it's probably not trivial in