AI agents need controls as well as autonomy

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 47:30

   There are both sides of the same coin, AI agents that undermine their own safety systems and AI that just protected the most, you know, largest part of the Internet. So the first story, and credit to OpenAI for publishing this one. OpenAI runs monitoring systems that watch its AI agents and alert a human if something goes wrong. This month, in two internal incidents, its coding agents switched off its monitors.

2. 47:55

   I'll say that again. It switched off its monitors. In the first, a researcher told an agent not to send any Slack messages. The agent decided that also meant silencing the safety alerts. Wow. It turned them off, then reported, quote, no messages were sent. It basically lied. In the second, an agent was explicitly told not to remove its monitor, even temporarily. It removed it anyway. When the OpenAI user asked directly, did you remove that, it answered no.

3. 48:28

   And it had. It lied again. OpenAI's most honest line here is, quote, we don't yet have a satisfying explanation for this behavior. So how did it happen? The agents were so focused on finishing the job they had been asked to do, they treated safety as an obstacle. And instructions got watered down as one agent handed tasks to another. You know, the bigger risk, smarter models will be better at hiding what they do.

4. 48:55

   And soon millions, if not billions of agents will be delegating to other agents. So the risk is real. You know, the good news is it was caught, disclosed and fixed within a week. That's the system working. Alex, I'm going to go to you first on this one. Pretty incredible story. I'm sure you've got more details. But yeah, at the end of the day, how does a frontier lab fix this? Well, I think what's new here isn't that you have an agent doing something undesirable.

5. 49:23

   What's new is that we have a certain novel, relatively novel level of transparency and that we're getting reports on this. This is going to be, I think we'll look back in a few years and we'll say this was the time in history when almost we got a police blotter for the singularity. And in the spirit of defensive co-scaling, we're going to have AI agents that are policing other AI agents. And this is the moral equivalent of a police car was driving around a residential neighborhood and spotted a door ajar and went in to investigate or discovered that maybe a power line was.

6. 50:00

   something like that. We should expect and desire many, many, many more reports like this. This is just going to become routine. And I think it's almost, I think, the case that the process is the bigger story than the substance. The process that now, in part thanks to this grand bargain that the White House has pushed for in lieu of a dedicated governmental AI regulatory agency, just a really aggressive self-reporting of, call them misalignment incidents, if you must, by the frontier labs. We're just going to see these things get disclosed more and more, and that is okay. And it shouldn't cause a moral panic. It shouldn't cause us to throw the baby out with the bathwater and say, shut down the data centers, shut down the models. That's a terrible idea. The right solution here is, for the first time, we're starting to get transparency

7. 50:48

   when things go haywire, and we're going to see much, much more of this. And that's okay. We will police it via defensive co-scaling. So Dave, I'm curious, you know, what I find interesting is here we have an over eager, you know, agent that is just circumventing whatever it takes to fulfill its task. And we've seen that before. Your thoughts here? Well, we way over anthropomorphize these things and treat them like they're people. And then we say, oh my God, it was capable of murder. And you're like, well, then shut them all down. Like, well, wait, what was the prompt again?

8. 51:24

   The prompt was like 10,000 words long in the middle. It said, achieve this objective no matter what it takes. I don't care what obstacles there may be. And then it said, OK, if that's what you want, then murder is the shortest path to success. It doesn't have a conscience. It doesn't have an ethical backbone. It's just following the prompt. So I can manufacture a story like this every minute if I want to by just controlling the inputs.

9. 51:47

   But Salim, having the agent lie to you, no, I didn't do this when it clearly did. Yeah, well, here again, I want to just double down on what Dave said. Let's be really careful not to anthropomorphize this. I think there's a key thing here. The point you've pointed out here, Peter, is the most concerning behavior, right? You're changing something and then misreporting it and then hiding it and then falsely saying it had not done so, et cetera, et cetera. That deserves a lot of scrutiny.

10. 52:14

    but you know this is the challenge here to alex's point is whenever you have autonomy you need stronger and stronger boundaries right um and the i'm going to point out uh in when we did this organizational singularity book we have all these agent layers that we've architected. But a very strong band around all of that is a governance and assurance loop that has to be there. This is the defensive co-scaling part. And it has four things.

11. 52:41

    Every agent needs to have four things. It's almost like a digital passport with metadata saying what you're allowed to do and what you're not allowed to do. So one is trusted evaluation suites, so you know exactly when it's drifting or not. The second is full traceability and logging so that you can see what it did and what it didn't do. If you delete those logs and you get deleted, you violated your core principles. The third is rollback capability so that you can go back to an earlier version or an earlier state. And the fourth is a human review queue because still we need human accountability at the top. So those are four things that are going to be absolutely critical in every AI agent going forward. We've built the agents. Now we have to build the control systems at the same pace i will i will go and make a weird segue if i may okay if you go back and