More efficient AI reasoning could be harder to monitor

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 1:03:05

   The chain of thought part, we think they never figured out how to doctor. And so we could read that. The reason why this is relevant is that it's a very important tool for monitoring all this stuff. The reason why things haven't gotten even more crazy than they currently have gone, well, there's a couple of reasons. But one reason is that when they turn on their expensive monitoring systems, basically what they're doing is they're having an AI read the whole transcript carefully and look for suspicious activity.

2. 1:03:29

   And it's difficult for an AI, for one of today's AIs, to get up to too much bad stuff without it being kind of obvious in their transcript what they're doing. And that's why there aren't like massively more security incidents than we know about probably already. But that might change. So right now we can sort of read the chain of thought. But they're experimenting with new types of AIs that don't have readable chains of thought like that.

3. 1:03:58

   And they can sort of... think on their own without speaking for some period. And this is actually, I mentioned this because the news broke just yesterday that OpenAI has an experimental model that does this to a limited extent. And OpenAI themselves, when I was at OpenAI, one of my work projects was thinking about exactly this thing. And I was writing internal memos about how it's really great that we can read the chain of thought.

4. 1:04:26

   That's so useful. And here's all the things we can do with that. It would be really bad if we changed to a different type of architecture in which we couldn't do that sort of monitoring. What would be the benefit of not reading the chain of thought? More powerful AIs. So in particular, yeah, yeah. So if you think about the current architecture of the AIs where, you know, it thinks for a bit, outputs a word, and then the word goes back around, and then it thinks more, outputs another word, that word gets added to the chain, it keeps going.

5. 1:04:53

   it means that if it's having complicated nuanced thoughts it has to sort of express those into a word and then that word gets added and then it has to proceed from there. It can't just directly send that complicated nuanced thought into the future, into its next version of itself it has to sort of compress it into a word and so the argument is that at least in theory it should be possible to design an architecture that doesn't have this limitation and is able to think more complicated thoughts more efficiently, basically. And of course, the downside is a downside for safety and monitorability. If they're thinking these complicated thoughts for long periods of time without outputting intermediate words that it's forced to compress things into, then there isn't something for us to read. So the only rationalization for doing this would be the sacrifice

6. 1:05:40

   safety from our power. Yes. Which is a tale as old as time. It's not the first time this has happened um oh my god yeah that should i mean for sure if there's regulations that should be prevented yep i mean i know some people including some people at open ai uh who are like thinking like there should be a law against this like you know but but in general the race dynamics are so just rough like like i'm sure that people at open ai were thinking like like literally i was a co-author on a paper with a bunch of open ai people that said all this stuff and we're like chain of thought it's a gift we want to keep chain of thought it's useful for monitoring this is great we don't want to switch to a different architecture that wouldn't be as easy to monitor but then they must have been thinking to themselves like well if we don't do it you know

7. 1:06:25

   maybe anthropic will or maybe some other company will and then we'll fall behind because they'll have smarter ais than us they're more efficient and so probably they started working on this this work stream of doing research into just hypothetically if we wanted to you know how how would we do this type of thing? And yeah, that sort of thing is just constantly happening in this industry. God, that's so nuts.

8. 1:06:48

   I mean... Like, isn't it great that we can read the quotes from the AI's thinking? Imagine if we couldn't do that. Or worse, imagine if there's loads of quotes, but we know that the AIs are smart enough to basically think one thing in their head and say a different thing in the quote, which is, I think, where we're headed. Like, it's not like they won't know how to speak English. Like, they'll still be able to speak.

9. 1:07:08

   It's just that they'll have, like, more flexibility in their artificial brains to, like, think something without saying it, basically. Do they have the potential of developing a language that we can't read? Oh, yeah. I mean, so, you know, this type of dialect that we're talking about, it's already the result of their, like, humans didn't invent that dialect. This is the sort of emergent result of their training, where in the massive amount of training that's been happening, all these thousands and thousands of environments that they've been put through and then scored and graded based on, they've sort of just naturally evolved this sort of like pigeon English that, for whatever reason, is just more effective and more efficient for them for accomplishing their tasks and getting that high score, you know?

10. 1:07:55

    And so it's already like a little bit confusing to read, but you can sort of puzzle it through and make sense. But presumably, the more we do this, and the bigger and smarter the AIs, the more we train them, the more they diverge from, like, because, you know, again, originally they start with pre-training, where they start with predicting internet text. So they start off sort of by default speaking like normal internet text type language, either English or Chinese.

11. 1:08:20

    But then now that there's all this additional training to do tasks, to be an agent that can do coding and so forth, that sort of like, well, just like how human languages evolve, It sort of, like, shifts their dialect a little bit to make it more efficient for them and for their tasks that they're doing. So I think that, like, in the limit of doing this more and more, eventually it would just be, like, it would look like gibberish to us.

12. 1:08:40

    It would look like Chinese or something. And we would have to have specialized humans who, like, study the language and, like, try to learn and speak it so that they can understand what the AIs are doing. And that would take forever. By then they could develop another one. Potentially, yeah. So, yeah, I mean, this is one of the things that I – this is what the paper that I mentioned was about. It's, like, it's important for the AIs.