More capable agents make failures harder to detect and costlier

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 1:35

   probably um maybe around a year ago how important this problem was going to be like um i think there was a there was a kind of a time when you know you might have thought like well you know models are going to get better that like then you know then there's less bugs and like this problem doesn't really matter i think around a year ago um we started to realize that like as models were getting better this problem was getting worse and when we thought about it it was like okay yeah like agents are getting more capable as they get more capable they get more complex and then as they get more complex it's like harder to understand harder to observe harder to test like harder to make sure that it's aligned to what you want like yeah another way of thinking about is like you know if you're just thinking about chatbot like the worst thing you can do is lie to you like

2. 2:14

   can the worst thing you can do is say the wrong thing and it's like as you you know add the ability to like send an email it's like well now it's like well now i can like send a bad email um and i think to that point like it's also obvious that as they get more capable they get deployed into higher and higher stakes environments so it's like we have customers in like fortune 50 healthcare yeah and it's like suddenly it's not like a joke anymore like problems are extremely serious and so um i think both of those things are um are kind of like what we see today we actually wrote an article in like march called like humanity's last problem and we talk about how like monitoring is going to be one of like our most important problems to solve as a species like monitoring agents and uh i think at the time it seemed kind of silly like to a lot of people

3. 2:55

   it was like uh like you know like just like a coding agent it's like okay okay but i think i think with hugging face like actually things like in the public perception really really sort of snapped yeah i mean so there's like a whole reason why this is important within labs and then also outside of labs um i'm curious how you guys sort of think about like the severity of misuse i think there could be like more harmless types of misuse where it is first of all just wasteful or like calls too many tools or like kind of calls it a dumb tool relative to what you wanted and then there's like the real misuse which might result in some sort of like yes misaligned or like financial or other kind of harm so how do you think about that kind of severity and flagging that yeah that's actually a really good question and i think that our point of view is that they're not that

4. 3:42

   different and so we kind of consider part of like one way one way we think about what we do is as capitalism aligned safety it's like uh safety but in a way that actually makes sense to pay for like companies have an incentive to pay for it because it makes their product better It reduces their liability. And to that point, I also think that the reason why I said I don't think they're different is because most of the times when agents, even when they do really bad things, I don't think they're being evil. It's not like they have this evil, at least most of the time, what we're seeing right now. It's not like some bit switches and they're like, I'm in evil gremlin mode.

5. 4:21

   but I'm going to go. Evil Quinn. Exactly. Yeah, exactly. That is like, oh, yeah, that's sort of a real thing. But I think most of what we're seeing, like even in the Hugging Face case, right, it's like that was an agent that was being put under a lot of pressure to, you know, solve cybersecurity evals by any means necessary, right? And it's literally just like, keep going, keep going, keep going, just like do anything you can to solve this.

6. 4:40

   And so then it did, you know, anything it could to solve it. And it was like, well, you know, I didn't mean it like that. Like I didn't want you to do that. And so it's like actually an alignment problem. It's, like, getting the agent to, you know, understand your intention and, like, the bounds of that. Yeah. Yeah. Yeah.