Palo Alto uses a harness to measure AI-assisted development

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 11:33

   This is something we've talked about, productivity being one of the most difficult things to measure. How do you prevent that? Do you have to, do employees have to say that, you know, I used, This is AI generated, this is non-AI generated. Like, what's your method? Absolutely. Great question, right? I mean, this has been a ramp. I mean, the last, I would say, three, six months, we have actually put a lot of focus on our SDLC harness.

2. 11:59

   And the harness is the way people can come into it. And that's the way we are watching where the spend is. When somebody is outside, we can see, like, for example, in IT, I have a harness that actually connects to both our first-party applications as well as third-party applications. Anybody using AI has to use through the harness. I also know if they're outside the harness where the spend is and we nudge them back into the harness because when I put them in the harness, I'm not just talking about coding.

3. 12:28

   I'm tracing from PRD all the way to what they're releasing in production, right? Today, I would say in some technologies, it's a little harder to trace it, but I'm able to get the visibility for over 85, 90% of the work that's happening. So I look at it as three things. One, what is the quality of code they're producing? How secure it is? Two, what is the velocity of outcomes that we are generating or release outcomes?

4. 12:56

   Three, how efficiently we produce that work? For a simple example, I mean, the market is heating up, security is heating up. We are actually, you know, doing a lot of M&A and the new product introduction for Palo Alto has gone skyrocketing. I mean, for me as a CIO and my organization is responsible for NPI, every single innovation that comes to the business. And we have gone 3x NPI over the last six months, and I have not increased even a single headcount on my software engineers.

5. 13:30

   But are they good product? Like, this is the question. Are they good product in a sense? Help me understand the sense of, like, is AI producing good code? Yeah, yeah, no, no. I mean, I guess what I'm saying, so product launches are one thing, but have the products meet the goals that you have set. Yeah, yes, yes, yes. Because it is human in the loop, Akash, right? I mean, I don't believe in autonomous agents yet.

6. 13:56

   We are not at the place where, you know, having autonomy fully, especially when you're looking for outcomes and the bigger things. I always have the code is generated. There is always a human review before the release. And also there is actually a user testing before the release. This also brings us to what we are launching today with XDR1 because the XDR1 will give me the endpoint security guardrail.

7. 14:19

   So I can be a lot more taking higher risk on because I'll have a full visibility of how the AI, I mean, we already rolled it inside, by the way, but we'll have a full visibility of how the agent is behaving and how I can have the appropriate guardrails and kill switches to protect it if it's going wrong. And it's also connected to our, you know, SOC XIM solution, which has a way to respond in an autonomous fashion. To go back here and question, I think it really comes down to having the high risk or higher value things, putting human in the loop. And the second thing is really looking at it from the point of you have the right security guardrails. And if you need to go down the path of saying no, we have a quick way to respond to it.