Cyber agents need domain expertise and layered safety checks

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 27:14

   And I think that's what accelerated it. You know, that was pretty much a firm stamp. It's common. Yep. What about the hugging face incident? I'd love for you to talk about the warnings from that. I've given that a lot of thought. I mean, I'm certain at OpenAI, like, oh, or at a regular, they were like, oh, we could have done this and this, and it wouldn't happen. You know what I mean? So they've already figured it out.

2. 27:22

   it's been my experience in every technical modality shift we underestimate the adversary's capability and in this case we underestimated the model's capability because you know when you really read it post facto ah they could have stopped that you know yeah and um they could put guardrails on it some deterministic things and uh and i think they realize that now but i think when you're in a race it's almost like a lunar landing race right the air is and you have r&d people and they're doing the work to create models in a way where even those ceos are like we can't slow it let's get the government to help us slow it you know that means you can't even control your own innovation I have views on that, but we can take it another time.

3. 28:03

   And so when you have, and I get that, R&D people are like chasing that innovation. And it's really hard to package them with then like security, experienced security people that have the skill sets to cage that thing, you know? And it's hard to marry those two up because the security people don't understand the AI as well. And the AI people don't realize. One of the things that we did in our model, I mean, make no mistake, Armiden has made the beast that we're all worried about.

4. 28:29

   We've made a model that attacks. We've made many of them. We have a system that attacks production networks and it's highly successful breaking in. Well, is it safe? Well, our guys instinctively knew we got to have obviously a secure, you know, we got to have a hypervisor. We got to secure this thing. We got to lock it down host-based. We have to have a proxy. It knows the proxy. It's proxy where that's fine.

5. 28:53

   But then our guys did something and even I was like, nice job. They passively, surreptitiously look at every single prompt done. Do we like it? Do we not like it? And the majority of the time, if we kill an agent, it's probably nothing to do with safety. It's that the agent's wasting money. Yeah. You know what I mean? So kill it. It's off on a goose chase we've already done or don't want to do. But there were so many layers of validation that the agent was doing the right thing.

6. 29:20

   And the other thing was, assume every layer of your security will fail and you have to have deterministic rules that eliminate certain activities. But what I did learn reading those incidents, it does take domain expertise to secure agents behaving in certain domains. You know what I mean? Yeah, it's a group. So I get that. So like without a cyber background, I get how you're going to make, you're going to test something, go, oh, didn't think of that.

7. 29:48

   Yes. And you would have had to have an experienced team look at what the evals looked like to say, you know what, it's going to do this and it's going to do that. So that's why Armadin, we combine the exploit developer types and red teamers with the AI folks because our evals most of the time are made by the red teamers. Right. You know what I mean? They're the ones that understand this stuff. And we created 20 full kill chains at Armadin that humans have done in the real world, period, at different victim sites and our experienced operators have done when testing networks.

8. 30:21

   and we had no model go through the entire kill chains of more than eight. So that's where it was eight out of 20. And here's what's weird, by the way, we tested the open weight ones and the most advanced closed models, they all found eight. Yeah, really? So if you're, yeah, so it was all about just speed and cost. Of course, yeah. And the closed models were faster to finding exploitable risk, but that's coming down.

9. 30:44

   But we kind of let the open models run longer and they got to the same place. That's it. So in the cyber world, the differentiation between closed and open is not as great as in other domains probably. And it's compressed, yeah. I would say that's somewhat consistent in terms of like the capability gap at least closing a little bit. But that's interesting that their performance is basically the same.

10. 31:07

    From my perspective, seeing the charts from the team, all the lines ended up in the same place. When you're looking at, it was immediately time, cost, and then call it effectiveness or creativity. And they all ended up, at the end of all their operations, where they hit diminishing returns, they ended up in the same place. Not on cost, though. Yeah, not on cost. Yeah, that makes sense. Yeah, that makes sense.

11. 31:31

    So it's a decent segue maybe to talk about what kind of money. models you guys are using and then what role do you think the lab companies play in the future well and i don't even know if i finished answering your last question other than domain expertise on security is going to have to work with the ai folks because i did read the you know the the meter publication and i was like well these guys are ai people but i'm not sure they've done a lot they don't have the domain expertise yeah and that's going to happen again the modality shift because here's one as when cloud started emerging you know i was running a bunch of incidents we were responding to breaches for a living at Mandiant and we had to learn what's a cloud breach look like, right? We now have to learn what's an AI breach look like, how much data is that

12. 32:12

    anthropic or the model companies that are being leveraged to do the attacks? What do you wish they logged and how they will change behavior as well to have better audit trails, better forensic capability. So it's early onset to the technology and we all clearly have to mature into it in a way where we have the accountability. When these things go rogue, you can say, here's what happened and when and it shouldn't be like two weeks of forensics to figure it out i hate to say it like we log every single thing our agent does source ip address time and date and what it did you know so you got to go backwards and replay these things and i think they've learned lessons the hard way and they're probably way better today than they were even three months ago at open ai and anthropic and testing these things and whoever had a regular labs they're looking at this going

13. 32:58

    okay we got to tighten up a little bit here and and i get they were surprised because they underestimated it and we've and we've done the same with human adversaries it is the weirdest thing my whole career everybody underestimates the top tier of what you're up against right because you don't have to see it every day yeah you know and and you don't want to fear the the boogeyman as they say under the bed so you don't want to have fud but you still want to respect the technologies that you're creating and test them in a way that is meaningfully guarded.

14. 33:27

    Got to cage the beast, David. Do you know what I mean? You do. And I would argue, cage it too much, release a little bit. Find the line because if you do overly deterministic, here's the art form to safety running AI, at least in cyber on offense, is you want to leverage the intelligence of the frontier labs to do stuff, but not do the wrong stuff. So you need classifiers. You need a model that looks at everything going outbound, everything coming inbound. We've created that with people and humans going thumbs up, thumbs down. That's good. That's bad. You got to train it, classify and look at it. But if you get too deterministic and disallowed too much, you're probably not leveraging the creativity of it. You let it out. So it is a gray area. And that's the problem with that gray area is that's why no one would ever say we're 100% certain we're going to get what we expect.

15. 34:18

    you know period it's it's it's a battle we even have but we have yet to have an issue because we can put a human in the loop or we have classifiers to say human needs to decide this we don't know what the hell just happened you know what i mean yeah yeah so if you're inspecting everything that's come you know every prompt to the to labs everything coming back and something comes back and we don't know what the hell it is pause escalate judge yeah what do those great security operators look like inside Arminen?