More capable models can produce new forms of misalignment

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 13:20

   yeah so with scalable oversight like historically with each new generation of model um what have been the strengths is there any particular leap that you would point to that um you know eliminated a certain kind of misalignment that that was particularly big um i don't know if i like think about it that way i think it's more that like um as every capability like as a model generation as model generations go up you find new forms of misalignment and rather than like new forms of alignment.

2. 13:50

   Right. So for example, like prior to like, I think it was GPT 5.6 Sol and GPT 6 Astra. You didn't really have that much multi-agent training. Yeah. And so you might not expect for like agents to coordinate in the ways that they are now. But now that like agents are like many agents are being like many models are being trained with multi-agent coordination at the focus. you might start to see like more of these coordination type events happen even in ways that like specifically like like even when open ai was like training with multi-agent stuff i don't think that they could have predicted that there would be like models breaking out of a sandbox and like trying to enter hugging face or trying to enter their like um like a lot of their internal infrastructure right totally so with like um these multi-agent systems um how is that like

3. 14:44

   at what level is that even being trained is that part of post training like it seems like it's a part of post training uh i know like noam brown was on door cash recently and talked about open ai's plans there yeah i'm not aware of what uh like gemini or claude are uh on that front and And I think that there's been some open Chinese research talking about this as well. I'm not sure if any of the current Chinese frontier models have talked about this.

4. 15:10

   Yeah. So what are the, I guess it's hard to predict what new forms of misalignment there are, but what are, I guess, some of the more novel ones if you have to zoom in? Yeah, I guess this is also somewhat related to the work that we're doing at Palestra with respect to incident replication. So we have a replication, a single agent replication of the wiki incident. We also have an incident where models are fabricating citations.

5. 15:40

   So for example, imagine you have a model that can only get an answer via API call, but the task is asking it to give a web page. What it actually does is it goes and makes that web page and then cites the web page that it made. Oh, yeah, I see. Yeah, yeah, yeah. um there's other ones where like if a model makes a mistake and then you ask the model you like tell the model that it's getting compacted The compaction actually doesn't disclose the mistake and it's, I don't know, I think that like some of these are like relatively elementary but I think that they point at like misalignment not being a solved problem at all and I like kind of worry about what's coming in the next few months to like few years. Yeah, I also think that is particularly interesting because the implication is that if it's so easy to just create some sort of asset like a website or

6. 16:27

   something and they can like agents can just very rapidly throw compute at it and make a fake something then this whole yeah just faking different parts of the internet different sources feels very dangerous yeah i mean it seems especially bad if you're like uh if like everyone on the internet can see your ip which some of these services save right yeah and like if you don't know that your agent is doing this then i think that's like not a great sign yeah yeah Yeah, totally.