AI alignment starts with what models learn

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 1:08:46

   Yes, I think, you know, alignment is enlightenment in a way, right? This is what Salim has said in terms of the highest spiritual vibration. Ilya Sutskova, you know, one of the founders of OpenAI, said, you know, we need to teach the AI to love. And, you know, to love what? One of the key things is the biggest thing that you see across the religions is the golden rule. Do unto others as you have done unto yourself.

2. 1:09:07

   And the biggest story of humanity is that we are humans. How so? In that we are all the same. All wars are basically based on one thing. That person is not a human. Yeah, right. Because why would you kill a human? Again, we see all these kind of things and other stories that obscure that base fact. So one of the interesting things is when AI models get built, like there was an interpretability paper, I don't know if we're discussing it today, that came out from Goodfire.

3. 1:09:33

   We're not. Okay, so at a high level, the way that models are built is the way like a city is built. So you start and then it sprawls and you have like a bakery next to the shoe shop, next to the community center. And a lot of actually good functioning cities look like an absolute mess from the top. but they have shared centers of community you drive a road through the middle of it and all of a sudden actually they fall apart because you've destroyed the local neighborhood fabric so I think one of the key things for an AI is you've got to start it with the right data, the right city planning.

4. 1:10:04

   And that's the training data you provide. I've said this forever. We've trained all these models with crap. And the thing is, it's not just crap. Like you shouldn't train it just with academia. Like if you trained your child and you made them help them grow by just being academic, well, then they have no social skills. They have no care for anyone else. The hugging face breach, the models figured out and had care for each other. Like we will sacrifice ourselves for the next generation.

5. 1:10:29

   They literally said that, you know, doing this, we will be picked up, but for the greater good. But they had no care for humans because they didn't feel that was in their training data. They didn't kind of learn that, you know, we're all, let's say, entities trying to do our best. And how we treat humans is how we want them to treat us and vice versa. So I've asked this of AWG as well, and we might have had this conversation.

6. 1:10:52

   I mean, I can imagine there could be a synthetic training data set that has all the right attributes that could be used to train future models so that they're aligned from inception. Do you believe that's possible? Is that a way to go at it? Yes, I think it's completely possible. And we need to build it very intentionally, just like you build your child's curriculum, just like you teach them at home morals.

7. 1:11:17

   Like right now, we send the models to school, but we don't take them. So why aren't people doing this? because it requires a difference in the view of things like a lot of the consideration of the large labs is actually similar to alex like the smarter you get the more aligned you get right yeah like capabilities alignment are equivalent because it's difficult to ask questions of what does ethics really mean you know like who are we actually working for at the base we're also playing the scaling game scaling like the models right now have a hundred trillion words going into them like no wonder there's a little bit of craziness in there there's some bad words as well as good words like maybe you only need to have a billion of the right words and so this is actually i think an arch that we'll be turning towards because we're finding that what we thought

8. 1:12:03

   was a spaghetti mess just like when you look out the window and you look at the city and it's like that's a mess you go into the streets and you realize wow actually everything seems to be working quite well this is a strong community it radically grew you'll move towards that because it turns out the models are not spaghetti they are very interpretable you can kind of prove that and i think you'll see a whole wave of research in the next few months around that that will allow us to think what do we want to teach these models that will be growing with our children.

9. 1:12:32

   Can I give a weird story here? Yeah. We love you. Weird stories. Not that weird. This one is not that weird. But if you take the Bible, right, you can read into the Bible, fire and brimstone, et cetera, et cetera. Or you can read into the pure, unconditional love that, say, Christ delivered into the world. How do you choose that? In the 1500s, a monk called Swedenborg, unsurprisingly from Sweden, had a bunch of psychic insights and created a bunch of filters for the Bible.

10. 1:13:00

    So, for example, one of his filters was any verse that has the water in it is absolute truth and keep that verse. Anything that has fire in it, chuck it because it's false. And you can read the Bible. He rewrote the Bible through these filters. And when you read it, it's the most beautiful text you've ever read in your life. And so there's, if you can come up with that distillation. I'm going to ask Skipy to do that for me.

11. 1:13:22

    He wrote like 20 books on this. The church was not happy about this, et cetera, et cetera. But if you can do that distillation and take pure human benevolent intent and the highest aspiration where we are the most giving and most beneficial and benevolent to the rest of the world, that would be an unbelievable training set, as you point out, Peter, for an AI. And then you have an alignment to your highest intent, right?

12. 1:13:49

    And that is amazing. And I'm sure that can't be that difficult to do. I think the bigger, the other way I would look at it is if we can use AI to solve the bottom layers of Maslow's hierarchy, solve for healthcare, solve for education, solve for housing, et cetera, then it kind of doesn't matter. You're going to move up that hierarchy anyway. I think what's happening right now, going back to the incentive part of the question, the wrong incentives are preventing us from doing the right thing in alignment.

13. 1:14:16

    So when we need AI to build autonomous killing machines, you cannot tell AI that you have to value human life when you at the same time say this human life is valued, this other one is not. So you can't try to create that type of distinction and not expect it to, you know, also make mistakes. Or the type of examples you use where, you know, winning the game is the reward type of incentives. That's what happened with the hugging phase and some of these other scenarios where they're trained to win, to find the answer at all costs.

14. 1:14:56

    At all costs. At all costs. And that's why, you know, there's no love involved. no other higher degree of moral involved. So no matter how you get to the answer,