AI labs lack a safety proof as incentives push them to scale

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 10:47

   So when I started working on AI safety, I thought I can actually help solve it. My goal was to solve it for humanity to get all the amazing benefits of superintelligence. And what was this year? When was this year around? Let's say 2012, maybe around there. But the more I studied it, the more I realized every single part of the problem is unsolvable. And it's kind of like a fractal. The more you zoom in, the more you see additional new problems you didn't know about.

2. 11:15

   And they are, in turn, unsolvable as well. Oh, boy. How is your research received? Like, when you talk to people that are, I mean, have you had communication with people at OpenAI and Gemini and all these different AI? I go to many conferences, workshops. We all talk, of course. In general, the reception by standard academic metrics is very positive. They've great reviews, lots of citations. Nobody's, like, published something saying I'm wrong.

3. 11:47

   But there is no engagement. I basically said I'm challenging community to publish a proof. Give me something, a patent, a paper in nature, something showing the problem is solvable. Typically in computer science, we start by showing what class the problem belongs to. Is it solvable, partially solvable, unsolvable, solvable with too many resources? Other than my research, we don't even know what the state of a problem is, and I'm saying it's unsolvable.

4. 12:12

   Prove me wrong. And when you say it's unsolvable, what is the response? So usually I reduce it to saying you cannot make a piece of software which is guaranteed to be secure and safe. And the response is, well, of course, everyone knows that. That's common sense. You didn't discover anything new. And I go, well, if that's the case, and we only get one chance to get it right. This is not cybersecurity where somebody steals your credit card, you'll give them a new credit card.

5. 12:41

   This is existential risk. It can kill everyone. You're not going to get a second chance. So you need it to be 100% safe all the time. If it makes one mistake in a billion and it makes a billion decisions a minute, in 10 minutes you are screwed. So very different standards and saying that, of course, we cannot get perfect safety is not acceptable. and again stock options financial incentives they continue to build it and they continue to scale and make it more and more powerful i don't think they can stop if a single ceo says i think this is too dangerous my lab will no longer do this research the whoever's investing in them will pull the funds will replace them immediately so nothing's going to change We'll see.

6. 13:27

   sacrifice their own personal interest. But overall, I think the company will continue as before. So this is logical. And the problem is, like I said, when I've talked to Mark Andreessen and many other people, they think this is just fear-mongering. We'll be fine. This is worst case scenario. We'll be fine. It is worst case scenario, but that's standard in computer science and cryptography and complexity and computability. You're not looking at best case.

7. 13:57

   I'm ready for the best case. Give me utopia. I'm looking at problems which are likely to happen. And it's not just me saying it. We have Nobel Prize winners, during award winners, all saying this is very dangerous, 20%, 30% P-Doom. This is standard in industry. 30% is what surveys of machine learning experts are giving us right now. So what is worst case scenario? Like how could AI eventually lead to the destruction of the human race? So you can ask me how I would kill everyone. Sure. And it's a great question. I can give you standard answers. I would talk about computer viruses, breaking into maybe nuclear facilities, nuclear war. I can talk about synthetic biology, nanotech. But all of it is not interesting when you realize we're talking about superintelligence, a system which is thousands of times smarter than

8. 14:48

   me. It would come up with something completely novel, more optimal, better way, more efficient way of doing it and i cannot predict it because i'm not that smart jesus that's exactly what it is we're basically setting up uh adversarial situation with agents which are like squirrels versus humans no group of squirrels can figure out how to control us right even if you give them more resources more acorns whatever they're not going to solve that problem and it's the same for us And most people think one or two steps ahead.

9. 15:22

   And it's not enough. It's not enough in chess. It's not enough here. If you think about AGI and then maybe superintelligence, that's not the end of that game. The process continues. You will get superintelligence creating next level AI. So superintelligence plus plus, 2.0, 3.0, it goes on indefinitely. You have to create a safety mechanism which scales forever, never makes mistakes, and keeps us in decision-making position.

10. 15:48

    so we can undo something if we don't like it. And it would take superintelligence to create a safety mechanism to control superintelligence. At that level. And it's a catch-22. If we had friendly AI, we can make another friendly AI.