Banning self-improvement would require invasive surveillance

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 11:08

   recursive self-improving super intelligence uh there are definitely a lot of you know safety alignment and political considerations here like some politicians have talked about just banning recursive self-improvement for example um anthropic and open ai have talked about the need to not do recursive self-improvement until we can make sure it can be done safely so what thoughts do you have on this yeah so i think if if your ai keeps breaking laws then maybe you should slow down. But that doesn't mean everyone else has to slow down with you.

2. 11:45

   I have heard and seen that those two companies also actually talk about how much AI is helping build the next version of AI. So they're also proud of how much they're stepping in that direction. At the same time, sort of philosophically, are opposing it to some degree. I think you obviously have to do it safely. And that's why we have incredible researchers that have worked, like I mentioned, rainbow teaming, and so on, on those exact issues. I think reward hacking and the flip side of reward engineering is extremely important and needs to be addressed. And we're doing that. And I think we have a better handle on it than maybe anyone, though, obviously, we don't know all the details.

3. 12:30

   of all the other labs. I also think that to outlaw recursive self-improvement would require a totalitarian surveillance state, the likes of which we've never seen, and would basically have to lead to like a thought police. Because if you just ask your little like model on running locally on your laptop, hey, like see if you can improve yourself and like a very simple form of recursive self-improvement is just a prompt like update, right? You just say like, oh, something you've learned, you then add it to the prompt. It's sort of the most basic form of self-improvement. And it's not full recursive self-improvement in its entire glory where you have access to the entire model lifecycle. But it's sort of that last bit that is the cheapest.

4. 13:20

   Now, if you asked your LM, oh, like running on your local laptop, make yourself a little bit better as I ask you these questions. Keep remembering facts that I asked you. This could be considered self-improvement. And now to like basically know that that person is doing it and try to outlaw it, you'd have to have incredible and very scary amounts of surveillance, which I think would make is a bigger downside than any of the downsides that people are worried about.

5. 13:48

   And so I hope people will be reasonable and, you know, just like with other open source projects and Linux or cryptography and so on, realize that it doesn't make sense to outlaw intelligence. That's ultimately what a lot of that regulation boils down to. I think it absolutely makes sense to regulate applications of intelligence, but not intelligence itself. And it's often very helpful to drop the artificial in regulation thinking and just think about regulated intelligence.

6. 14:21

   You don't say, oh, this person's really smart. We should put them into jail just preemptively because if they were trying to commit murder, they could probably get away with it because they're so smart. That's not how the law works. The law works when you actually commit a crime. And so I think likewise in AI, we, of course, have to regulate self-driving cars. They should be tested and certified. Of course, we have the FDA check drugs that were created by an AI and have them go through proper trials.

7. 14:48

   Of course, we want to have an AI neurosurgeon robot be very carefully tested before it gets applied to a human brain. Of course, we apply laws in the applications, but we don't regulate intelligence in the abstract.