AI creates value by making harder work possible

Transcript

From the transcript The Pragmatic Engineer publishes with the episode.

1. 1:03:04

   like number that you can sort of like pin to. And my theory had been, so measure all of the things that we think are positive or negative indicators of something is going well. Releases, test coverage, number of PRs, everything. Just cast a wide net. My initial formulation on this was the wisdom of the crowds idea. No individual can estimate how many jelly beans are in the jar, but you get 100 people and you'll get real close.

2. 1:03:35

   So we do a like, am I hot or not? Which would you prefer? One release per week more or X tests, right? And you run a bunch of iterations of that game with your relatively experienced engineers to work out the linear trade-offs between all of those factors. To figure out what matters more. Right, to figure out what it is. You probably want to de-dupe some things because they're just going to be the same stat in a different package.

3. 1:04:07

   But find the ones that have some non-linearity. And then... Mush that all together, right? Now you have something that sort of gives you an aggregate, like this is kind of our productivity number. And Adobe is doing something along these lines. Slightly smarter than my initial formulation. But like, then you can watch that track up or down and, you know, over time for a given population. What I've realized is that won't capture everything in this post AI boom.

4. 1:04:41

   And the reason that I came to that was a paper that I put out, the Develop, Deploy, Operate paper recently, where one of the points in there is there's four forms of impact in tech. There's product success, the stuff everyone talks about and everyone else is better at than I am. Like, that's not my job. People buy a product or use a product, whatever. Net promoter score revenue, like all of this stuff, right?

5. 1:05:08

   There's... Compute efficiency, right? Like, how much are you blowing on AWS? Oh, yeah. Right. Or if you're shipping. I'll send yours all too well. Right. You know, or like even like the Android team has a tradeoff for if you can reduce the size of an application download by a megabyte.

6. 1:05:26

   It makes a difference for many metrics, yes. Right. You know, and so like all of those things, like that is a form of impact. Then there's engineering productivity or we phrased it as engineering capacity. Because if we save you 5% productivity gains, capacity gains, you don't get to spend that somewhere else. You just have more capacity on doing more engineering. Or having slack time for innovation or picking up when something happens, that kind of stuff.

7. 1:05:55

   Right. But they're not like dollar fungible. Exactly. You don't get money in a different account. And then the fourth form of impact, and this ties back to the AI thing. Fourth form of impact is changing the game, right? Strategic innovation, like research makes impossible things possible. Security makes incredibly bad things impossible, right? But if you change the nature of what we are doing, the cost structure, etc., then that's a different game.

8. 1:06:24

   And the problem that I think that we see with AI, even with a clever technology, You know, index metric approach to trying to capture the gestalt of productivity. If you're not looking for the value of a given change is substantially different, then there's no way you'll ever be able to capture that in the signal, right? And we don't have anything that says the value of a change of this size has changed.

9. 1:06:51

   So for instance, like it is entirely possible at this point. I hang out in security spaces and do a lot of low-level C++ library stuff. You can write a JPEG parser in a couple weeks with whatever additional features you need. That would be insane to say that sentence out loud three years ago. Okay, so what you're saying is...

10. 1:07:18

    What none of the measurements really capture is the possibility, the impact, the effort it is to do a massive innovation, which right now with AI is happening. I'll give you also one example and tell me if this is what you mean with it. There's a friend who works at a CTO at an engineering company, a mid-sized engineering company, they're a niche industry. The CTO, they have been looking at release frequency, the usual metrics, talk with their engineering team, are they happy, are they not?

11. 1:07:55

    And of course, now they're adopting AI as well. They're seeing that, well, people are more efficient and just privately see this person was telling me like, look, like I kind of know that for a fact that we could probably do the same amount of work with like half the amount of people and then we could be cashflow positive. First, there's a morale thing. So he's like, also like, he likes the people.

12. 1:08:14

    So it's like, I don't, I'm not sure I want the emotional package on me, but we can see some people are picking it up faster and some slower. And we kind of left it at that. A few weeks later, I'm talking with this person that says like, you wouldn't believe, but we're probably going to be cashflow positive. I'm like, how? Like, did you go ahead with the cut? He's like, no. What happened is one of our engineers came to me and said, like, we license this component.

13. 1:08:37

    Think of a library, like it's a binary, but that was written by a vendor in the space that has a high performance thing that they use in their software. It's a core part of it. And they have a revenue share where they pay like, I don't know, like 5, 10, 15% of their revenue for this component. It's so important. It's obviously licensed, closed source, but it's really important and it would take them Years of a five person or 10 person team to write it and it might or might not work.

14. 1:09:02

    So it's not worth it. Then you came and said like, what if, what if we just gave the specs? Because they have all massive amounts of tests around this thing, a test harness. They gave it to AI. AI wrote a component to them. And at first they thought, well, maybe we'll open source it. But then they told the supplier and the supplier said, like, it's a huge business for them. They're paying them to not open source it.

15. 1:09:27

    And to give them the implementation, which they can re-license. And suddenly, the game was completely changed. Like if this person would have gone ahead with, let's say, the layoffs, this engineer might have just got it. And no one might have known that the game Right, exactly.

16. 1:09:48

    And so I think that this is the insight that tries to bridge the gap between I have not seen a research based number that says productivity post AI is up more than like 20 or 30 percent. But then everyone talks about like these 10x sort of experiences. I super don't think it's actually 10x, but yeah. Yeah. But like, how do you bridge that gap? And I think it's exactly this thing of a lot of the value of AI is not in doing the same thing faster.

17. 1:10:19

    It's doing a harder, more valuable thing that you wouldn't have even thought about. And if your productivity measures are the logical equivalent of counting how many widgets are coming out of the factory... And then you start producing super gold-plated deluxe widgets. There's no way to measure that or track that over time because we just don't have the tracking of what's the value of this component or feature.

18. 1:10:43

    And that also matches with talking with senior engineers that are like, yes, we're up 70% on amount of code. We're up 30% on outages. Number of features is the same. The thing that we can't know for sure is, are the value of those features up? Because how do you do that?