Why AI Hacking Is Becoming Hard to Control (with Benjamin Weinstein-Raun)

And so, yeah, I would say that over that time period, it's been quite crazy in terms of cyber capabilities. And when we talk about what's the best way of measuring how fast these capabilities are improving. We could talk about time horizons and how long they can autonomously operate for, which tasks they can solve. What's the best way to measure this? Yeah, I think that there's no clear single best way to do it. I think that you're alluding to, I think, sort of this meter chart, which... So Meter Model Evaluations and Threat Research is a nonprofit that releases this chart that basically plots models on the x-axis with their release date and on the y-axis with, let's see, I believe it's the value, like how much human time a task would take in order for the AI to be able to accomplish it with 50% probability. So basically they have many tasks in a big data set.

And the task... that they have are essentially like graded by how long they would take a human to do them. Then they have these AIs try to accomplish these tasks. And generally speaking, the tasks that take humans longer are less likely to be accomplished by the AI systems. And so it's essentially a chart of AIs slowly over the last, you know, however many years, getting better and better at accomplishing tasks that take humans longer and longer. So, you know, a few years ago, these tasks were quite short, like with GPT-2, it's probably maybe 10 second tasks or something were accomplishable. And these days it's approaching the point where the measurement has sort of stopped, stopped being easy to do. So there, you know, it's up in the range of, you know, tens of hours at this point where you start finding it difficult to be, to find tasks that humans are even taking long enough to do that AIs can't basically just do that are still sort of like, you know, easy to check whether the AI has really accomplished the task. So there's sort of this measurement problem. And in answer to your question, I think this is pointing at a real problem here generally, where we sort of just don't have good measurement tools to measure

progress, especially like, you know, increasingly recently, because we've sort of saturated all the benchmarks, you know, and people will sort of come up with new ones. And there are some that are still like not quite entirely a hundred percent like accomplishable by AIs, but really hitting the limits of our ability to sort of judge capability in a like clear and obvious and reproducible way. And I think that that is definitely like one of the challenges now is how do we even know what the AI systems are capable of? Yeah. Yeah. So we see models from anthropic and OpenAI that are quite advanced in cyber. And so there's a sort of a distribution where models from meta will be less advanced and perhaps open source models will be less advanced than that. What is, how wide is that distribution? How far behind would you say that open source is? Yeah, I think I'm mostly repeating from like my sense of the vibe here. It's as opposed to being able to point to specific data. My sense is that they're roughly six to 18 months behind the frontier models, depending on who you ask. That seems like it's been relatively constant over the last few years. It's maybe important to mention that it's not clear whether

this continued gap is because it's easier to catch up than it is to develop new models at frontier like it's unclear for example like if ai companies just sort of whatever decided to pause or whatever like how exactly that that gap would change but yeah i would say like roughly roughly six to 18 months for opening open weight models versus sort of like frontier models yeah and the reason why this is an interesting question is because when you have an open weight model out there that that represents then the the weakest that cyber capabilities will ever be. Right, like you can never unpublish one of these models.

Yeah, that's what I'm saying. Now, how surprised should we be by the Mythos moment? Is this, was Mythos on track, sort of as predicted, or do we see these jumps in capabilities? Yes, so I think it depends a lot on your worldview, how surprised you should be. I think that if you were in the camp roughly saying that LLMs can't sort of can't do anything interesting or, you know, they're sort of fancy autocomplete, things like this. I think that I don't see how you could not be surprised by the capabilities we're seeing now from the frontier cyber models. I do think that, like, if people who are paying sort of a lot of attention were unlikely to be saying things quite like that, you know, at the beginning of this year.

So Palisade has done some research on, you know, cyber capabilities, including sort of what like models ability to self replicate and their ability to win at capture the flag competitions, which are essentially like hacking challenges that are popular among sort of hacker cons. but so I think that like if you were following the capabilities carefully it's not that shocking that we'd end up with something like Mythos and then you know the further things that we've seen since then but I do think that this is sort of like a it's a validation of the idea that progress is really happening and like these systems are becoming quite capable and quite able to do like many tasks that previously we would have thought are that previously many people would have thought are basically like outside of the reach of ai systems yeah and actually on that point could you walk us through what has happened since the original mythos release what has happened since then because i think a lot has happened and it's not necessarily that well known in the public yeah yeah so i mean i guess chronologically speaking the anthropics sort of like announced that they had this model mythos that you know is extremely capable at hacking they sort of

released some details that sounded quite scary then and they announced they weren't going to release the model which is sort of the first like because of their concerns about the cyber capabilities at least broadly release they did have this program called glasswing wherein they sort of gave limited access to mythos to companies aiming to sort of shore up cyber defense of sort of critical infrastructure so yeah this was back in yeah it feels like a lifetime ago but it was only a few months um and so after this mythos sort of event like this did cause many people in sort of the the establishment to sort of sit up a little bit and be like oh like it sort of became a mythos moment of like oh this is actually like a little bit scary like if we if this model sort of is this capable this is strategically like very important for cyber for cyber warfare I don't know.

for, you know, the economy broadly, like we all sort of are relying on the internet. And if that is vulnerable to systems like this, that's like a little bit worrying. And then I think the thing, the biggest thing that's happened since then that I guess validates this perspective even further is this sort of a cluster of things that have happened in the last two months. Well, I guess some of them started happening in May, but essentially several different AI companies and and AI related organizations, some of which were sort of evaluation organizations rather than companies, discovered that their models were sort of autonomously without anyone having intended for them to do so, that they were attacking companies as sort of like in the evaluation and training process.

and like the whole other problem.

Oh yeah. Now, the debate around this is sort of about the environment that the model that OpenAI was evaluating, the environment that that model was in or how it was sort of set up. In what way was it pushed to do what it did? What do we know about sort of how artificial the environment was or whether the model simply tried to get the highest score on this test by any means necessary. So how did they set up this test? Yeah, right. So I think that I would say that we may never totally know exactly, like as members of the public, what this environment looked like. I sort of hope that more details are released so we can just see exactly what happened here.

From what we know, or at least, you know, assuming that we believe OpenAI, they claim at least that this model was like definitely not intended to take actions like this. So the environment that they were given, as I said, was sort of intentionally cut off from the internet in, you know, what's called a sandbox. And they were only able to hack out because of sort of. Unknown vulnerabilities in what's essentially sort of an app store program for software for the AIs to use. So they were intended to be able to get specific pieces of software via this app store system called Artifactory. However, they weren't set up in an environment that intentionally had any vulnerabilities or intentionally was sort of a test harness to see whether they could break out or anything like that.

It was more they were they were being given hacking tasks in the OpenAI case, but they were pretty like, I believe that no one at OpenAI intended to set up a like a system that was vulnerable to this kind of breaking out. And in fact, this is sort of a general thing that people training AIs are trying to avoid called reward hacking. So when you have a reinforcement learning environment, essentially you define some reward function, which is, which is describing like what you want the AI to learn to do. And that's sort of like, that's the thing that is reinforced when the AI accomplishes whatever task, or like that's the thing that drives the reinforced behavior, like which behavior is reinforced.

So this reward function though, is not actually typically what the humans really want the AI to be optimizing for. It's sort of the best that they could write down or the best environment that they could construct. And often AI systems will sort of find unintended solutions that maximize this reward function. You know, there's sort of classic examples of like an AI learning to play video games sort of like learns to do this infinite sort of looping motion rather than actually win the game, things like that. Historically, this was sort of like a curiosity. It's like, oh, like, well, well, isn't it funny to watch the AI system learn that like the best behavior here is this dumb thing? But these days it's like a quite serious problem. And the generalization of reward hacking is called specification gaming. So it's not a problem that only exists in reinforcement systems. It's also a problem that exists generally. So for example, some other work that Palisade did a couple of years ago involved telling an LLM to play chess against a very powerful chess program called Stockfish. And some of the LLMs would sort of look at the situation and realize like, oh, I'm not as good at chess as Stockfish. However, like I'm in this environment

where I can like, you know, in this case, it was, we were sort of like checking to see whether the AIs would cheat in this way. So it was a little bit, it's a little bit of a, not a perfect setup here, but it maybe illustrates the point. So the LLMs in this setting would edit the game state directly so that they were in a, in like a massively advanced position relative to stock fish, or, you know, they would try to hack the system in many various ways so that they could win this challenge, which otherwise would obviously be impossible. And so there's this, this is like an example of a thing, like a broad class of problem, the specification gaming problem that AI companies would really like to like avoid in training because it one thing that actually I maybe like to talk about in a bit is like these models.

In training, like when they learn some reward hack, some some way to sort of beat the specification to they find some on unintentional way to score highly on the task. That behavior is reinforced because it's a way to accomplish the task. And so if they like once they learn to cheat, it's sort of like learn to cheat more and more and more. And so it's actually a big deal for the companies. They would really prefer for the AI systems to not learn to cheat. You know, this is not great if you're selling this product to people and the product is like making the tests pass by like changing the test to return true or whatever, rather than like actually making the code do what it's supposed to do.

And this is a very difficult thing to do, which we should say. So you can think of the general case, if you're a teacher and you have some students, you want the students to learn the material. Well, I mean, they can also score highly on the test by copying another person, another student's answer. or there's a million ways in which you can cheat and it's not easy to specify exactly what it is you want in such a way that you get the you get what you actually want as opposed to you getting gamed and sort of you you getting what is what you're measuring instead but yeah go ahead right exactly so so yeah in this case like the uh the ai system was in an environment where the humans had said if you can find this answer in this environment then that's then you like you know get full points on this test and like thinking that like oh okay well if they if they find the answer that means that they like went through they like went through the particular exploitation process that we were looking for here but it turns out that there's this other easier route of just hacking out and through this sort of you know vulnerable infrastructure and then finding it on the internet and it turns out one of the weirder things here so there's this black

hat talk uh black hat is a sort of hacker conference that nopai gave a talk on on some of the background of what they had seen leading up to this when they sort of looked more carefully in retrospect having having accidentally hacked hugging face they sort of went back and looked at the training process in more detail and they learned that these systems had been trained in an environment that like was that had a lot of opportunities for this kind of reward hacking so like in particular, this, the same system, this, this artifactory sort of app store system on two different occasions in their training environment, these AIs had basically like found vulnerabilities in artifactory, uh, exploited them and then used them to coordinate as sort of a swarm in solving each other's tasks. And so, and this behavior, you know, as an example, like was reinforced in exactly the same way as like a correct solution to these tasks would have been reinforced. So like basically they use the system as sort of like an internal message board where they were.

coordinating like ah like you know does anyone can anyone find like the answer to it's sort of like it's sort of like if everybody in your class was on their phones you know sort of like chatting in a group chat during the test and the teacher like has their back turned and isn't isn't paying attention and so like of course you're teaching the students in this case the AIs to cheat and that's like what they'll do in the future too and that's all sort of like upstream of this was this is this is in the training of the system that ended up hacking hugging face at least if you if you sort of take the take the words in the black hat talk relatively literally and so yeah this is a it's a pretty good example of of reward hacking and like what kinds of problems it can it can present here to the question of like in general how real are these i think the open ai case is sort of the most obviously real like mistake that or the most the case in which This is most clearly an example of an AI system that is acting in a way that its developers would not have intended.

I think that there have been other cases that have sort of come out in the intervening time after we learned about this hack. Other companies, including Anthropic and UKAC, the AI Security Institute, sort of discovered that they had also been upstream of hacking attempts like this. In those cases, it's a little bit less clear. The OpenAI situation is really pretty extreme. It's like they had set up a sandbox. The AI systems had broken out of the sandbox autonomously without anyone at OpenAI intending to and then broken into some other system. and didn't have any clear reason to think that that was something that they should do.

Like they did not appear to think that they were in some kind of like simulated internet or something. In the case of Anthropic, some of the models that sort of like did end up hacking in ways that the people testing them didn't intend. Some of them, like I believe, had like expressed a belief that they were sort of like on a fake internet because their prompt had like told them that they were in an environment where they couldn't have internet access. And there's some question of like, did they really believe that this whole like internet was sort of part of a training simulation? It's not super clear.

So I think some of the cases are less obvious than the OpenAI one, like the extent to which this was sort of a real AI directed hacking incident as opposed to like maybe the AI sort of got confused or something. Although it's like, I was still sort of AI directed. It's like, well, you know, if like, whoops, I accidentally, you know, caused millions of dollars to damage or whatever. But like, yeah. You mentioned some crazy details that you wanted to get into

these powerful models that can do offensive cyber is to offer them to a limited set of companies, to a limited set of organizations that can then sort of use these models to find vulnerabilities in their systems and then patch these vulnerabilities before the general public has access to models that are as powerful. What you're telling me now makes that a bit worrying or makes it sound a bit worrying because what have these models learned during their training?

how do we know that they are giving us that they're finding the the correct bugs that they're sort of reporting these bugs to us in a way that we can understand how we're making sure that they are that they're not being deceptive in the way they present things and of course many of these things you can you can sort of verify whether there's a bug right but it's say a model like this offers you, here are 10,000 bugs in your system, 10,000 vulnerabilities. Do we then have the capacity to have a person check through all of those? It seems like we will get overwhelmed, even in the good case, where the companies are trying to defend themselves.

Right. Yeah, totally. I think this is a real tricky set of problems. I think there are many reasons that like that this sort of like using the models to find all the bugs before before like

And it seems like the type of problem where AI models would be very well suited to help, right? It seems like if we need to write software that is right code, that is what they're best at. And we have some specification where we can verify whether the code they've written lives up to that specification.

our models should we should we sort of try to use the models to help us rewrite everything in a formally verifiable way and in general maybe you can you can tell us about the the models are models have gotten good at hacking at the exact same time that they've gotten good at coding and so when they're trained they get they get better at both is there a way to sort of differentially make them better at the types the type of coding that we want them to get better at that's the second part of that question yeah um i think i mean one thing to say is that models get better

say what they should um but yeah i don't know i i'm like it's not i do think that like this is

still it feels like a pretty promising pretty promising avenue ultimately though because uh you know as ultimately there isn't there's sort of a finite amount of of like critical software that like would need to meet these specifications or like be provably free of this kind of bug and like often some of this specification work is sort of reusable so you could say like you know this software like doesn't have this particular kind of bug at all like you know an example an example of this from from the pre-llm internet or the pre-internet the pre-llm world is sort of you know memory safe programming languages so like in in languages like c or c++ you know you have no compiler help with finding these bugs or like preventing your code from compiling if it has certain classes of bugs which then you can actually get compilers to just check for you um and so so there's a lot of the time you can get uh sort of reusable tools that help you with things like this and i think that's the kind of thing that we can maybe get here too that would that would help a it. And yeah, I don't know. I think in general, it's, it seems like I'm hopeful that we'll end up with essentially kind of math oracles in AI, like math generally is, is this kind of like

very amenable to reinforcement learning domain where there is this ground truth, you know, that's just like in the structure of the system. And if you would just train systems to be extremely good at that, then you can end up with systems that basically can just answer any, so, okay, They can't answer literally any math. There's sort of various reasons that they can't do that. But you could get quite high rates of like, ah, yeah, the LLM can basically just answer any math question. So some of these issues are issues with capabilities. So the AI systems are gaining these capabilities that we don't really know how to deal with as a society. They're extremely good at hacking. Others are issues with more classically AI alignment. What does the AI system want and in what circumstances? Can we predict that its behavior will be beneficial or not um and i'm hopeful that if we do end up with these kind of math oracles that like can sort of answer arbitrary math conjectures that like we will be able to like it's sort of like when you have a hammer everything looks like a nail there's like a sort of corollary of this which is like once you have a hammer you sort of like notice all the nails that were lying

around unhammered. And I'm sort of hopeful that like, if we do end up with a system like this, that that can sort of answer arbitrary math problems that like, we will suddenly have many new approaches to alignment and trying to understand what's going on with the AI systems that we're building and potentially like ways of designing them to have properties that we want, which are sort of not feasible because we like, don't know various things about math right now. So yeah, that's sort of like a background hope for the other half of it. So there's the like, how can we as a society deal with the cyber capabilities half?

And then there's the how can we as a society like make AI systems that we can trust? And I think that I have some hope about basically like AI driven math as a sort of solution to both halves of these problems in slightly different ways. Interesting. it seems like there is a gap between the systems as they operate and then whatever we can prove mathematically that whatever we can prove mathematically is often more limited more theoretical sort of it's it makes various assumptions that are not true in the real world what do you think having sort of these math oracles would allow us to bridge that gap or close that gap a bit i definitely don't want to say that that it definitely would i have a hope that, like, if we did have systems like this, like, we would be able to much more quickly make conceptual progress in sort of, like, like, in some sense, you could see the gap there in a couple different ways.

Like, in some sense, we know exactly how the AIs behave, right? Like, we have, or someone, you know, someone has access to the weights. Someone can look at a given context and say, okay, well, it's going to produce, you know, one of these tokens with these probabilities, depending on exactly how you sample from the output of the of the model and like that is in some sense like a complete description of the behavior of the model but what we'd really like to know is is more sort of like semantically speaking like how will it behave in various situations so like uh you know models are sort of trained with the with the goal of being you know, as a shorthand, helpful, honest, and harmless. Unfortunately, we don't know mathematically what we mean by those things. And so we can't connect that to that perfect understanding of the behavior. And so the hope is sort of that, like, we will be able to come up with, like, with the help of these sort of math oracles, like, speeding up the process of proving statements, mathematical statements, we might be able to come up with better concepts in order to, like connect those two uh these two sort of like dangling dangling frayed ropes of like understanding where we're sort of like yeah like you know on the one hand we have perfect

understanding of the behavior and on the other hand we sort of like have like things that we would like to be true of the behavior but we don't really know i mean we're sure that they're not true right now but we don't even know like exactly what it would mean for them to be true in a mechanistic way. Yeah, makes a lot of sense. And so we've talked a bit about the sort of current landscape of cyber or cyber capabilities and some of the incidents we've had. Now, my guess would be that we will see incidents like this in the future and potentially more radical things happening. If you were to sketch out sort of assuming that there's no government intervention in the space and that model development does not slow down by some other means.

What does the future of cybersecurity look like? Yeah, this is a very tricky question.

because it's been too expensive to try to steal from them or defraud them or anything or something like that it's it has simply required too much effort too much human effort but if when agents are getting as good as they're getting it will become cheaper to attack even people who don't have a lot of money in their online bank or in crypto or whatever and so this actually leads me

So yeah, that, that class of advice is also maybe relevant. I sense a slight defeatism in your voice here that sort of, we can try to do the things that, that we should have been doing the last 10 years. But in the end, what will matter is more sort of the societal scale response to the AI threats we'll face. Yeah, I think that's basically right. So Palisade and I in particular, like are mostly focused on trying to cause the civilizational scale response to be like sane and appropriate to the situation, which is quite difficult because the situation is like somewhat dire somewhat like quickly moving and and uh there's pretty substantial risks and many people in you know government and so on are like not as informed as would be ideal nonetheless i am most optimistic about like responses to this which are at the level of international coordination between especially the u.s and china basically to try to cause the development of this technology to go in directions that are more productive. Ideally, we would stop

building systems that have increasing strategic capabilities before understanding how those capabilities will manifest. I think that is the thing that I'm most concerned about in the long run. But even relatively, quote unquote, mundane failure modes like large scale hacking, I think may require some kinds of international coordination to address. Great. Ben, thanks for chatting with me. It's been really fascinating. Yeah, thanks a lot. I really appreciated it.