Software engineering at Google & CS education with Titus Winters

And here's my favorite part, guided reviews, which is work in progress. Guided reviews is a new code review experience where the important stuff just pops out. The top of the review has a summary and diagram of the change. And the pull request is divided into chapters that walk you through what's changed. These blocks highlight the important changes in a way that's easier to understand and review. Oh, and when Dima is finished with all of this, he just self-approves the code and is merged into GitHub, all without leaving Linear. If you take one thing away from this, Linear is no longer just a place for planning tickets and organizing the work.

It's where all the work happens, creating the issue, kicking out the agent, checking out the preview, doing the review, and merging to production. Linear is the product development system for teams and for AI agents, and I'm loving this evolution. Check it out and get a six-month trial at linear.app.pagmatic. Titus, welcome to the podcast. Thank you so much for having me. I'm so excited to have this one. It's awesome to do this in person again. Yeah, it's a joy. So how did you get into tech? Growing up, did you know that you're going to do something with I mean, my earliest memories, I thought I was going to be a scientist, but like, you know, as a three-year-old, that means like lab coats and test tubes and I don't know what else, right?

My dad very insightfully knew that computers would be important. Like that was not his field at all.

He was kind of in carpentry and set design, things like that. So there was always a computer in the house. It was usually kind of a low-end thing of, you know, whatever we could scrape together and afford. And then I remember I really got my start. I think I was around 11 or 12. And instead of buying candy for everyone for Valentine's one year, he went to the bookstore. And I don't know what my brother got, but I got a book on LearnQBasic. And it was all kind of downhill from there. Like I spent probably the next five, six years like programming, teaching myself to program was my hobby. Occasionally I would wind up like meeting, you know, other kids my age that were a little into it.

We moved eventually to Las Vegas and I went to a magnet school that had a computer science program that is actually a really well regarded high school these days. I had no idea. And yeah, like just kind of self-taught initially. Pre-internet, back when you had to really fight for it. But yeah, that set me on my path, that book.

And then you, after high school, it was kind of obvious you went to college for it. Yeah. So I went to Harvey Mudd, which is a great school with a mix of, you know, tech and liberal arts education, the kind of place that'll teach you to build weapons and teach you not to use them, that sort of thing. And then I met my wife there, and she was a little bit more junior than I was, so I didn't want to move away. She was still in school, so I did my master's at UC Riverside, actually in computer science education. I did a kind of weird, I had a very strange graduate experience, but yeah, I spent four years at Riverside doing a PhD in CSED, kind of applied AI, educational data mining, computer science education.

So some of the earliest conferences on CSED research I published there. And my advisor was the department chair. He never generally ran grad students, like the research area was not his focus, but he was constantly the chair. He was this like charming, you know, social, political sort of guy that knew all the stories. And so my graduate work was a lot of learn all of the stories of how we got here and teach all of the classes whenever there was a hole in the Yeah, and we'll keep coming back to probably this, like this education, computer science thing throughout this thing. So we've talked this before, so I know.

But let's jump back. So like you now graduated from college. You were looking for a job. Rob? Yep. What did you find?

So first I did a sort of consulting group. Again, two-body problem. My wife went to grad school in San Diego. I did not get a professorship after my PhD. So I did this consulting stuff. I was at a small startup consulting group, about 30, 50 people. Mostly I worked for, indirectly for DirecTV, doing the embedded system stuff on set-top boxes. Mm-hmm.

And then after four years of that, we were done with grad schools. We were done with all of that.

I took the offer to do a Google interview, got the offer, decided not to go to the Bay Area, and they moved us to New York. So we've been here ever since.

This is when Google was opening an office here. The office was already open and reasonably big. This was 2010. So I think there were a few thousand people there.

And then in 2010, so Google was founded in 98, they went public in 2004. I remember that they started to really accumulate a global brand as like, wow, Google is the place to work at around like 2005 to 2010. So right around when you were applying, it was actually one of the hardest places to get into a bit like today, Entropic or last year, OpenAI or the You know, like in the mid-2010s Uber. How did you find the interview experience and getting an offer? Because it must have been pretty competitive at that point. It was competitive.

I was a little surprised that they gave me the offer. I felt like I didn't do that great. I remember... For some stupid reason, I really, really didn't want to spend a bunch of like precious vacation days. So I flew up from San Diego the morning of instead of sensibly like, you know. They would have paid for the hotel and everything. Yeah, I just didn't want like, I don't know. I'm an anxious person. I like my habits and routines and stuff. Getting up at five for a big interview is not a good plan, folks. Hot tip. So there are a couple of things like I remember one interview was like a relatively simple like clock math sort of thing.

It was technically camera angles on a Mars rover or something. And there was a point in there was like, I'm tired and having interview brain and this is just basic math. So here's a function that has the right inputs and outputs. And if I wasn't stupid this morning, that would work. And I just did it all in terms of that. And I think my interviewer actually really liked that approach of like, yeah, this is not the important part. Like, let's just talk. And so, yeah, eventually they gave me the offer and, yeah, moved to New York and started up.

And when you joined, you got the nuclear hat. Uh-huh. Yep, absolutely. And so what did you work on at Google when you joined?

So briefly I was on and like I don't think that Google had the best placement for new hires necessarily. Onboarding process that we say. That was a little rough because like I said the things that I'm interested in are like Developer efficiency, problem solve. Like I have a background in networks. I have a background in like API design, those sorts of things. They said, oh, you know, networks. So let's put you on this team that monitors the amount of traffic and where that is crossing the edge of the Google network, which is actually a databases problem. With networks as the topic of all of the data that you're doing.

I'm like, I don't know anything about databases. That was never my thing. So it wasn't a great fit. And then almost immediately after that, like two months in.

I want to push a little bit back here already. It's like, come on, you're joining Google, you had college, you had four or five years of experience. When you joined a company, especially a hot company, they said, all right, do X, and you're like, I'm not really there. Would you not want to say, let me just go deep into this to see what's important for the business? I am pushing back on you here. Yeah, no, please do, always. I...

Did what I could to learn, you know, from that team, for sure. There was a little like interpersonal tension in some of those things.

Okay, so it already started a bit like that. You could have had this nagging feeling. Yeah, like it was just like, yeah, this wasn't quite right. And then there was a like email that went out to all the engineers in New York saying like, hey, we're thinking about spinning up a new team that handles all of the old like C++ nuts and bolts infrastructure code. That was written by people whose employee IDs are like two digits. They have more important things to do than to handle like patch requests for all of those components. So we would like people to maintain that. And me and I don't know, nine or 10 other people like raised our hands and I was the only person that showed up to like the second meeting.

So I just kind of got onto that team. I get it. So basically, I think the context is just important because I think, you know, folks hearing, oh, I'm joining this team. Like, I didn't really like the first thing. I think the difference, the context that was missing is like, first of all, Google was growing really quickly. New teams were starting up left and right. And also like, Moving teams was kind of fine, right? And you were more looking for like, all right, like you knew that this team, both for what they were doing and also sometimes there's not a great people fit. It would probably not be the one where you're going to be staying for many years.

You did the best you can. And then when you saw an opportunity, you knew it was better for your strengths. You're like, let's go for it. So for you, C++, low level stuff. Yeah, I mean, back when I had been doing the consulting at DirecTV, there was stuff like build an API to do UPnP, like Universal Plug-in Plays. It was like local network RPC sorts of stuff. Remote positive call. Yeah, right. And so it had been a lot of people, teams were manually assembling like the XML structure for every request to like implement a single client call or something like that. It was like, we could do better folks.

And like, I was probably a little too clever in the way that I assembled all of that API, but it was like, yeah, it was a thing that I liked.

Was it more of an initiative of like, hey, Google has a bunch of C++ here and there, kind of like roots up, like, let's get together and initially in our free time, let's see if we can do something? Or was it more some manager type said, like, I'm actually Do you think of assembling a team, a full-time team on this? So this was, I think, like the last year before Google got actually pretty formal about you're embedded to a particular team and headcount is limited. It was a little chaotic. It's a little loosey-goosey about how those things work. I could actually be wrong about that because I was still, you know, new enough that the internals of how headcount works might have been just not clear.

But yeah, like moved over pretty quick and boy, that was exactly the place for me. All right, tell me about it. Why was it the place?

What kind of work can we imagine of, like, I mean, because it sounds like when you describe to me, like, taking random C++ stuff that people wrote at Google, like, that sounds like the version of hell for me, like, you know, like, as an engineer, especially pre-AI, right? Like, Understanding what another engineer at some point wrote who might have left and a team is using and it's working for them. It sounds like it's a lose-lose because if you take it over and you make a change and you break someone, they'll be mad at you. And if you do it and you do a great job, they'll probably not even think about it because it's always worked.

It still works. Great. Luckily, the fear and the safety net for making changes had been pretty well established a couple of years before this played out. Specifically, and this goes to topics that we'll get to in a second, I'm sure. Specifically, Google already went through the phase of, yes, you have to write tests. In 2010, this was not an obvious thing in the industry. Right. But like, I think the story is 2006, 2007, Search lost the ability to push a release on a regular cadence. Like, it was down to, like, we were trying for monthly and they were maybe only get once a quarter because...

Oh, so this is why it's Google obsession with tests and quality and... Uh-huh. So the core money-making product was kind of handcuffed, not being able to push around through 2006. Wow. And circa 2006, almost anything that you did had some tendrils in like the front end or search or stuff like that, the whole architecture and separation and like multi-app, multi-products, like world. That was still not, right? So if you were making changes to Gmail, there were still chances that you were making changes to search. And so everything was holed up. So the release engineers, they banded together and they simply just said, you will not merge this change if there is not tests, period.

We do not care what VP says anything. We do not care how important this feature is. You will not do that. We will roll that back. And as soon as you put a line in the sand, Even the people that are like, I'm very smart, how dare you ask me to write tests? Like, this is beneath me.

And then so basically taking over or taking on maintenance or even changing any library, C++ library, you had the test so you could have high confidence if you refactor these things, all the tests pass. Yeah. Either it works or... There was never enough test coverage. Right, exactly. And either way, it's a win. Because it's better if you expose the lack of test coverage. Okay, you have a crash today, but tomorrow or a year from now, you'll have twice as much traffic anyway. Right, exactly. And like...

It's one of the things that I believe very deeply is if you are held back from making changes to a thing out of fear, you have to go in and fix that. Like fear is not an acceptable basis for engineering. And I think it's in the first two or three pages of the book. I've said now for 10 years, you know, your code base is sustainable if for any valuable change that comes along, you are capable of making that change safely, right? You may choose not to make that change, right? That could be technical debt or, you know, different product priorities or whatever. But if you're incapable of reacting to change, then you're placing a high bet or high risk bet that the change isn't critical or isn't important.

And I hate to break it to people, but the security world has gotten real, real bad in the last few years. There is nothing that is not going to change. The strategic advantage of being able to change is incredible.

Is it safe to say that if I join a company, a startup, a large company, whatever company, I joined a team and I look at the code base and my first weeks, my first months, I'm just being afraid of pushing code changes because I worry this thing will break. I'll break production. That I'm working on a system that is just, let's say, poorly engineered.

It could be the system is poorly engineered or it could be there's some onboarding and mentoring that didn't happen there. So one or the other probably. One or the other for sure. And I think, and this goes to some of the educational stuff, like we... In undergrad, you know, education and like the way that we kind of portray what the job is like, there is a correct answer. There is right and wrong. It is not just the tradeoffs. And you're graded anytime someone looks at what you have done, right? And I think it's going to be, especially for new grads, it's really hard. It's really easy to get into a fear of like, I don't know if I did this right.

I don't know if this was the answer you were looking for because they've been falsely given this perspective that there's a right answer instead of, of course, it's just a tradeoff, right? It's at some point in the design space.

And I guess you're also saying that these new grads are also used to you do something that is a meaningful effort, typically a class or an exam, and you get feedback. You get a scale, you get a grade, you get a thumbs up, a pass, or a fail at the very least. And I know I should remember when I went to my first workplace, I should remember this really weird, like, join the company, build this project, and then nothing. And then I, you know, I do another thing and then nothing. And then I have a, I have a one-on-one with the CEO of the company and I'm thinking they're going to fire me because like clearly like I, I knew that I don't know, like middle of the day, like I was sometimes slacking off because I finished my stuff.

In hindsight, I was pretty efficient, but I just finished my stuff and then I was just browsing the internet. I told them I'm ready. And I was pretty sure like, Maybe I'll be re-imprated or something. And instead, I'll be like, oh, amazing job. Like you've onboarded so quickly. We're so happy with you. But the difference was that up until then, I've had years of getting direct feedback and it didn't connect to my mind. Yeah.

The other part of that is like the only time that a human is going to look at your code if you're an undergrad is to grade you, is to judge you. Whereas when you become a more experienced engineer, you're asking for feedback regularly, but it's just a check-in. Are we going in the right direction? Does this make sense to you? And the more of that feedback that you can get that isn't judgment, The more effective you will be. I'm like, oh, I hate that we've like given all of the new grads and all the junior engineers exactly the wrong instinct. So going back to Google, what is Abseil? How do you pronounce it?

Abseil. Yeah. So Abseil was starting about 10 years ago, shortly after the initial pivot to cloud, there was a motion to Hey, we have all of these amazing developer tools in addition to all this amazing production stuff. Maybe we should try to externalize the developer tools as well. And at the same time, there was also a little bit of Google shifting into Alphabet. So for both the developer tooling aspects at one level of like, how do we share developer tools amongst different companies within the Alphabet umbrella? And how do we share high value, like internal interfaces and APIs and, you know, tech components?

How do we share those when it's not one, you know, single corporate, one single company? And also not one monorepo, right? Increasingly, yeah, exactly. And so there was a bunch of work that was kind of the normal, like, okay, what does it take to make these developer tools like multi-tenant and like, you know, slightly less... Completely tied into that ecosystem. And just multi-tenant, for those who are not as familiar, what does multi-tenant mean inside of Google?

Well, so this would have been multi-tenant of other companies within the Alphabet umbrella, and then maybe extend that out to cloud customers. Oh, got it.

Yeah, so like a Waymo and the Google could theoretically use it, something like that.

Okay. So the largest refactoring history at that point in human history might have, you're not extricating. I don't think so.

I mean, I can't prove it, but I would be surprised. So how, okay. When you have this type, you were in the middle of this 10 to 12 people, you have this massive, massive Massive task that seems giant and almost, how do you go about it?

hang on, what if we told the AI, okay, let's do a thorough code review every now and then, but when it finds something, let's have it generate a lint rule. Yes, please. Which is cheap and And it will no longer take 30 minutes, the cost. And also it's just non-deterministic. It might or might not find it next time. That was frustrating. Like this person was like, look, when we know we have certain things. So now what they're doing is every now and then they still run some of these reviews.

And of course, but whenever they find any issues, then they fire up. Inference, AI, several agents, and it's like, all right, can you come up with an improvement in our deterministic linting that can help it? And the inspiration came actually from this person heard about how Uber, actually, I think it was, heard me talk about when I joined Yeah. Yeah. Yeah. I think it makes perfect sense. Like, let's try to learn our hard lessons only once, codify those rules somewhere to Germanistic.

Apply them equally and appropriately and fairly and leave the interesting parts of code review. Make sure that the parts of code review that we're focusing on are only the things that humans need to be doing, which was true then. I don't think everyone invested in linting and static analysis quite to that extent. They should have. And then it's equally true now. We can't review everything, but there are places that require this human attention that is well-spent.

Titus just made the case for learning the hard lessons once, then writing the damals rules and saving the human attention for the things that really needed it. That is pretty much the same way you want permissions for AI agents to work, which brings us to our season sponsor, WorkOS. WorkOS provides the auth infrastructure that B2B and AI-native companies need to sell to enterprise customers with SSO's Kim Arbak and now Authorization for AI Agents. WorkOS Airlock is intent-based access control. Most access control only asks who you are. Airlock also looks at what the agent is trying to do at the moment it tries to do it.

Clear-cut rules like no agent deletes cloud resources are run as plain code. For the fuzzier calls, an AI just evaluates the policy at runtime. You write those policies in plain English and Claude can pull the ones you already have out of Notion or Gyra and import them for you. Your agents, whether that's Claude or Devin, connect to a single access point and your MCP servers are behind this access point. Airlock is a new layer on top of your existing permissions, which means that an agent can never get more access than the person who invoked the agent. When something risky comes up, Airlock can escalate to human for approval, say over Slack.

It's the same idea Titus described. Write the rules once and spend human attention only on the calls that need it. You can request early access at workerwas.com slash airlock. I also want to mention our presenting sponsor, TurboPuffer, a ridiculously scalable, fast and cheap hybrid search engine built on top of object storage. How often do you get to see a rewrite playing out in real time? With the Teepuff team, we finally can. They are completely redesigning their storage architecture from first principles to make search faster, cheaper, and more reliable at scale. The new system called TurboPuffer v3 is not yet in prod, but it already passes CI.

And when the rewrite started in September, it was not nearly as performant as the current production version, v2. The T5 team had been perf-mapsing query plans until they are fast or faster than v2, doing all this in public all the way up to the v3 production launch. And it's so cool to see their progress. The new system went from being 10 to 100 times slower across most metrics to being several times faster on things like cold vector search, order by attitude, or hybrid search. The V3 system is still slower in areas like full text search, but only one to two times slower versus a hundred times slower when this perf maxing started.

And the TPF team is sharing exactly how they optimize things in their development logs with things like group count aggregation, compressed column encodings, batch scoring, and many more. A lot of these wins are basic foundational work right now, but the team will be sharing some very cool systems work over the next few weeks. But hang on, why is the team doing a full rewrite? It's because search is changing, and so is TurboBuffer. TPUF started as a vector database, but it is no longer a vector database. It has evolved into a high-performance database for search. The V3 discards the vector primary index, and all indexes will share the same columnar storage system.

Every query will run on vectorized query engines, which is something you don't usually see in search databases, but which will result in both very fast search and just generally a very fast database. The TurboPuffer team is keeping a work log of their journey and you can follow along at turbopuffer.com.

It's a blast to see how it's being built. Speaking of hard, hard lessons and documenting them, can we talk about Software Engineering Google, the book that you're quite well known for? How did the idea come? How did it come together? What was your motivation to start with?

So this was a little bit the tail end of, hey, maybe we'll ship the developer tools. Maybe we'll need to explain why Google's developer tools are the way they are. Because otherwise it'll be a little bit of like aliens came down and you're trying to give you something that doesn't make any sense in your world. Part of the reason that we had the hubris to like tackle that sort of task was because we had been doing this refactoring at scale, you know, everywhere. And when you... Run into everything that any engineer thought was a clever solution to their problem. It meant that my cadre of weirdos have run into so many, oh, yep, I can see how you thought that was a good idea.

Don't do that. Like more than anyone else out there. You're talking some of the most kind of battle-scarred people in the industry, just because you're working on these large-scale systems for such a long period of time.

And it's not the production battle scars, it's the like... Testing and local code design battle scars. Everyone else in the industry has all this production experience and my job ends at Merge. I know more about development and API design and test and keep the code base as an artifact working than most, but it does mean... I'm kind of an outsider in that sense. Well, interesting enough, I mean, the book is very, very thorough on how it gets to production, but it does not cover the operational part, which I guess is the SRE book. But there is an SRE book, which, well, then this book would be like twice as long.

How did you pull the book together? Because you wrote some parts of it. Other folks inside of Google wrote different chapters of it.

So it's very much in the spirit of the SRE books. Yeah. Like try to get the expert on or an expert on Gavin Topics to contribute something useful. And then the three names on the front of the book, myself, Hiram Wright, and Tom Mantrek. Hiram and I were kind of the technical leadership for that. And Tom is plenty technical, but has been our tech writer friend for many, many years. So he handled a lot of the like editorial, structural, organizational stuff. But Hiram and I had been the tag team of... Hiram generated millions of edits and sent them to me for review, and I had the permissions to review almost everything in the codebase.

And so everyone knew us as those two guys that clean everything up. So we had those scars, and so we led that. And then the trick was finding the experts to write something high quality. Some of those were great from the start. A lot of that took a bit of ghostwriting stuff. The whole task was actually almost two years. Certainly not full-time. We were at most probably putting in a quarter to half-time most of that time. But yeah, we wrangled together way too much text from way too many people. I don't think I realized just how much we overshot until I saw a printed copy of that book. I was like, wow, that's huge.

Yeah, and even today, I do recommend the book because it gives a really good understanding of how the Google at the time worked. Obviously, there's been evolving since, but how a company going from zero to building important production systems came up with a bunch of stuff that might not make as much sense. And you talk about things like Like monorepo, like the stringent code review process, how there's a readability process at Google. And maybe we can talk about that. Google is the only company I know where for code review back in the day, they didn't just have code review like an engineer who owns this part of the system and understands, reviews it and says it's okay.

But Google had a readability review. What is readability review?

Their world-changing accomplishment is almost always the result of a spark of inspiration followed by a heroic team effort. A great team makes brilliant use of its superstars, but the whole is always greater than the sum of the parts. But creating a superstar team is fiendishly difficult. Yeah. I don't think I wrote that chapter. It's in the book that you are a co-author of, so...

I wanted to reflect on this, especially with AI, because now I'm starting to hear a lot more people say, yeah, in the past we talked about a 10x engineer up to one, up to 22, 23. Don't get me started on that. So I just wanted to Ask you how the concept of the team holds up as we have more AI tools and the concept of a standout team. Yeah.

So I think regardless of where you fit on the spectrum of like AI believerism, I think the team is still critical. Like if you believe that the agents are the only answer in another direction forward, We're still managing that team of agents. And I've heard an increasing number of voices talking about it as the problem right now is we need to make every engineer into an engineering manager for a team of agents. And we don't have the training or skill set or background to have done that. Like we're asking them to do a thing with no scaffolding and guidance. The alternative is you think of the AI and the agents as your tools, your IDE, but it is still you in conjunction with the team of humans around you.

It's still team behavior. I think there are certainly differences in the social dynamics and things like that between you and the agents. Which I'm going to go out on a limb and say the agents don't have feelings. They definitely do not. Right? Should not. Let's hold that line for a bit. But yeah, like I think understanding clear communication and like being able to write things down and get it out of your head is still, you know, a substantial important thing. Like I... Yeah, I think it is still unquestionably is still teams at the root of it. Software engineering is a team sport, period.

Also, I guess just one like pretty like naive way to just prove that this is probably still true is if we just look at some the leading AI companies, the leading AI labs, the Entropics, the Open AIs. When I talk to people there. Yeah. And that team, when I met them, you can tell they gel. They work together. They trust each other. At some point, Boris will be on holiday and you will not know because things just happen. You said the important keyword, trust.

A substantial part of this is the psychological safety aspect of it. The other thing, which I think is natural for the AI labs, but hasn't been nearly as dominant for the rest of the industry yet, Is you're learning as you go. And if you look at all of the research, you know, like Dora research on stuff or any research on productivity or team efficacy or any of those things, like at root, it always boils down to culture, psychological safety and the the permission to learn. It's the permission to admit that I don't know something. It is the incentive to learn from each other. It is the incentive to share your mistakes so that the team only has to do that once.

Your lint rules or SRE postmortem culture, every effective structure and organization in tech. Is actually about information flow and learning. And I think a really great perspective on that. I love Kat Hicks, the psychology researcher for tech. Like her book, The Psychology of Software Teams is beautiful. And like she talks about, you know, these things a lot, right? Similarly, there's a platform engineering at Google book that some of my friends are putting out as early release chapters on O'Reilly. I might have written the chapter on education for that. Very surprisingly slash not surprisingly.

Right. But yeah, like education is the powerful route for like, let's make this mistake a minimum number of times. Let's codify what we've done. Let's give people permission to admit they don't know and to grow because it's not a one-off game. It's an iterated game. And you'll do better over time if your team is better next time.

I want to go back to psychological safety because back about five, 10 years ago, we kind of talked so much about it in enduring leadership side circles. This was pretty well understood that, you know, A good team feels a safe team. If you, for example, make a mistake, you cause an outage as an engineer, like literally you push a code and that code causes outage. We should have a blameless postmortem where we don't say that, hey, Gergay pushed this code and he caused the outage because that's now, I mean, I feel terrible. But let's talk about, for example, how could have we avoided this at a system level?

Why did a test not fire? Why was it rolled out immediately when he was a new joiner on the team or she was a new joiner? Maybe some onboarding could have helped, yada, yada, yada. How do you define psychological safety and why is it important? Especially just playing devil's advocate. Like when we're talking about smart engineers who know stuff, like, is it all that important to us? Because we should know our thing, right? I'm just like, I'm kind of removing my emotional hat here. Yeah, we should know our thing, sure. So I would say, and it's been a little hard to like, We could give you a 15 question, true, false.

Or maybe Likert scale, but like a 15 question assessment for the team and predict 30% of whether or not you will hit whatever your goals are. That's insanity. That is more powerful than any other signal. Like, okay, so you think that we're all engineers and this is all just computers and tech and emotions have nothing to do with it. Find me anything that's even remotely that powerful predictor of success. It's a people problem, like period. Right. And going back to. Right. It's it's not a single shot game. It's an iterated game. I regularly cite Fred Brooks and the Mythical Man month essays. Right.

And in the surgical team, like the surgical team is my least favorite chapter in that book because it's structured as if. We're only ever going to do one project. Get the best person to do this. Have everyone else build the tools and do the documentation. It's very hierarchical. Right. And it's very like power dynamics mad. Yeah. Right. But it might make sense if you're only ever going to do one task. Instead, we're playing like in a game theory sense. It's not prisoner's dilemma. It's iterated prisoner's dilemma. There's a different strategy. And the strategy is find the most junior person that can do it with supervision so that they learn and grow and you still get a decent outcome.

And that next, like the higher level people can solve something that they don't know how to do yet or that no one else can tackle. This is the only way to structure it so that growth happens. But in order for that growth to happen, people have to be willing to admit that they don't know a thing. And that brings you straight back to psychological safety and learning culture. If it is a culture that highlights information sharing, learning, and growth, you will succeed.

And you mentioned that the AI labs probably take this for granted. Because they don't know what they're like. Well, I'm going to paraphrase it. With the AI labs, there are a few things going. First of all, business is booming. They are growing. Revenue goes up. Customers goes up. Funding goes up. Headcount goes up. Their teams are only ever expanding. They're in hypergrowth. We've seen companies like Netflix and Uber in hypergrowth. This is even faster. Your equity also goes up. So you don't need to worry about things like performance reviews because performance reviews are typically important where like how much bonus I'm going to But when you're at a place that goes like this and your equity goes up, your financial part is taken care of just by focusing on helping the company succeed regardless of your titles.

Second of all, because they're building new technology and they know no one knows, so they just tell Okay, I don't know. I'm learning from you. Here's what I learned. So it's just all natural. Whereas compared to a company that might not grow like this, they might still be using, let's say, AI. But now there's less of an incentive to say, I don't know, because there's a performance review coming up. You want to look like this. You don't want people to say like, well, this person says, I don't know a lot. Your bonus will be lower. You might be put on a performance improvement plan because you're now at a company that every year lets go of 6% of people.

And your goal is suddenly going from like, I just don't want to be there. I want to hang on to this job, but it's now hanging on. And you also, you're now very of maybe your teammates because now there's an internal competition. You want to Hog that project. And so this is a huge difference. Yeah, it's huge.

Like if you get into the mode of believing that the team outcomes are what matter for your own success, then it is much easier for you to realize like, okay, I don't have to have this one shiny task. Like the team will be successful by virtue of someone doing this. And it might be my turn next. If when it breaks down into that like individualism and like, I don't trust the guy across the way from me or, you know, right shoulder to me, like even though we're on the same team, we're scraping by and like, you know, fighting for attention and success. And that's not a recipe for innovation and growth.

Like that is a recipe for stagnation and just bad.

Yeah, we've seen that most of big tech has had it, especially in 2023. Yeah. Speaking of Adobe, so you've moved to Adobe a few years ago. What are you focused on at Adobe?

So there I'm very senior, I see focused on developer experience. I happen to say this is a polite fiction. Developer experience is the business of productivity. We just know enough not to say that we can measure productivity. It does not give everyone...

Unlike McKinsey, who me and Kent have bonded over. Yeah, that's horrible. But we will link it in the show notes below. Yeah. So it's what do we need for tools and culture and all of the things to improve the experience of being a developer and make you more productive and efficient? Most companies spend more on engineering comp than anything else.

So today you've been focused on thinking about it and working on it for both at Adobe, but also to some extent at Google. What do you know today about developer productivity? How does this relate to AI, coding tool, output, metrics, all of that things? What is it that we can measure? What is it worth measuring? And what are things that are hard to know? Yeah.

I mean, we've taken it as axiomatic that it is hard, if not impossible, to measure productivity because what are the units even? I had a while back a sort of clever idea, and I use that as a loaded term. I had this idea that if we don't know what the units are, like economics has prior art for this, right? You have an index metric. Where you measure all of the things that might influence this and you come up with a way to effectively like basket those together. And then you have a unitless, but like directionally useful and useful over time,

One challenge that AI is giving is entering the market for new graduates. There seems to be expectation. Companies are very eager to hire senior engineers who know how to use AI, and they have taste and judgment. And we have these things thrown around. It's not deliberate, but whenever I talk to the AI labs, they always tell me increasingly that they look for taste, judgment, agency, these kind of things. Let's go back to education, to computer science education.

And, you know, you go back to 10,000 hours to mastery and Malcolm Gladwell. I think that's a Gladwell.

And like, yeah, like where are you getting the time? And if I think back of the boot camps that spun up in the from like early 2010s, they started to get really strong 2014 to 2017. What they did is a three month, three to six month Yeah, yeah, yeah. There was such a huge demand for software engineers that they got hired as a junior engineer level or almost at an apprentice level. But what they knew compared to computer science grads, they knew version control, they knew Git, they knew pull requests, they knew a database, they knew a frontend, they knew Chrome developer tools. So they actually had a bunch of valuable skills in only three months that the people coming out from the program in four years might or might not have had.

It was more of a coin toss. Yeah, exactly. I've done the math. The number of hours in a CS degree is roughly equivalent to like 15 months on average of full-time employment.

And that includes all of the theory stuff, all of the things that are questionably relevant and is similarly trying to take you from no experience to employable in either an industry or academic direction. And like that is... Compared to a six month bootcamp, right? Like that's that's not a huge gap, especially when one of those is laser focused on these are all things that are just immediately relevant. And one of them is here's some stuff that'll hopefully teach you how to think about these problems, maybe or less charitably. This was useful when I was a student. And also, can we just mention some of the most successful schools were actually outside of the US.

One of them, well, a lot of the schools are, of course, you know, the usual suspects are very highly regarded from the MIT's, the Stanford's and so on. But Waterloo in Canada, I was very surprised to see that a bunch of Silicon Valley companies have a high number of Waterloo companies. And I asked this on social media, like, what's on with Waterloo? They're up in Canada. Why is everyone hiring from there? So it turns out that Waterloo has this program where, yeah, they teach you computer science or software engineering. I'm not sure which one. But the main part is that every... They have dedicated placements.

By the time someone graduates, they will have had four placements slash internships at different companies. And because they have the relationship, this will often be the Shopify, the meta, this. So you will get an engineer who, three months times four, that's like a year working inside of these companies. And that is the balance. Yeah, exactly. I don't think it's just Waterloo.

Like a lot of the Canadian schools are- This was just one example, by the way. Yeah. But yeah, a lot of the Canadian schools for CS have substantially stronger programs on internships and co-ops in a way that a lot of those degrees run five years now. But I think it is a much stronger educational experience because there are some of those lessons, like I will fully admit, like there's stuff we did not put into the software engineering requirements. Because there's just no way to do that in the academic setting, right? There's some lessons you're going to have to learn the real way, like in the field, even though you gave the example of like doing a project.

I want us to do more projects, but I'm really nervous about there being one project course in most degrees because you know what no team has ever looked like? A bunch of people with exactly the same level of background, right? Like it also gives the wrong taste about this is what teamwork is like. I'll take it. It's better than not doing it. But. Well, we have what we have.

And this system will hopefully with people like yourself and others will hopefully start to also change or just adapt to the times. But AI is right now here. And one. Big question, more experienced engineers are thinking about. We've learned to build software pre-AI. We know the fundamentals. Many of us went through computer science, so we actually have a very strong theoretical understanding which we might or might not use some parts of it. But what about for new grads? Now that they'll enter a workforce of just today with just the capability that we have, what... Do you think the fundamentals are that they should focus on and what are the things that they should just rely on AI?

There's always a question of like, should you just ask AI for this or should you learn the basics, right? Yeah. So, I mean, I think learn the basics is an entirely loaded question because nobody agrees on what the basics are. And that's fair. Kent Beck said a couple months ago when we were last together something like everyone that's currently working has grown up in an era where there was a playbook. Like we know at least in theory the right way to do this and there's no playbook right now. Right. So we all have to get a little comfortable of not knowing the answer. And we certainly don't know the answer on what is still the basics.

The things that I think that the juniors should be focusing on one. The resilience to avoid just copying your ticket into a prompt and then trying to check in that result. Right. Like if you're if that is all that you are doing. You can be replaced with a very small shell script. Like that is not a defensible advantage for you being on staff. I think it's really important to use AI to better yourself and to widen your inputs, but not to immediately produce all of your outputs. There's a metaphor that I really liked from about a year ago at a think tank. We called it Andrew's Axe. And it's If I go chop down a tree with an axe, I do not say that it was the axe that chopped down the tree.

It's me that chopped down the tree. The AI, as much as we occasionally anthropomorphize it, is still a tool and 1979 IBM training manual. A computer will never be held accountable. Therefore, a computer should not be making management decisions. The AI is still not accountable. The AI labs are not going to indemnify us for the mistakes that these things produce. So you have to be able to stand up for whatever your name is on for those pull requests. And that requires a lot more rigor and not cutting those corners. And that attention to detail, I think, is really hard to train. Going back to sort of the undergraduate version of this, the scary story that I've been hearing from a lot of profs is, The best students, like the B, B plus and above students, they are better than they ever have been because they have.

Yeah, because instead of relying on a TA that is available, you know, a couple hours a week and is one or two grades above them, they have an infinitely patient, always available, incredibly well read, sometimes hallucinatory tutor. Right. At the drop of a hat. Right. They can always learn and explain and figure out. And they actually have the autonomy and the drive to use this. Right. And the and the metacognitive resources to avoid that. Like, I'm just going to copy and paste. Right. The very worst students are somewhat better. Because they used to not be able to do anything, but now they can produce slop.

Yeah, they just throw in slop, yeah. Which is a bit hard to detect, but I get what you're calling.

But everyone in the middle has kind of collapsed. They don't have that resilience. They would rather go to the party or play video games than do the homework. And so they're like, well, just this one time, I'll just get in the habit of doing this thing. And that slippery slope just gets worse. And so there are a lot of profs from decent schools that are like, Yeah, I don't know, man. Like, I'm not sure that anyone, like, below, like, the vast middle, everyone below a B, might not understand how code works at all. Like, the system is just not set up to be tracking that sort of thing.

Or at least giving them feedback, for example, alerting, those kind of things. Right.

And so a lot of the discourse initially was, how do we prevent students from using these AI tools? And I think that that's nonsense because it's echoes of, you know, we got high level languages and there's compilers. But now you're not going to learn how to program in assembly, right? The industry is using it anyway. Right. They're going to be asked to use, they're going to be forced, pressured to use it harder in some places when they get a job. Well, in fact, it might be, it will for sure be a red flag if a new grad says, like, I've not used these tools. Like, come on. Right. Have you not? Do you not want to work in this industry?

It's an open secret what we're doing. We're publishing it on blogs and everywhere. Yeah, exactly. Right.

And so the line of, well, we're going to never let you use these tools cannot possibly hold. I co-published a piece in Communications of ACM with a bunch of academics this month that I think the opening line there is imagine forward to 2040 and And we hold the line treating all use of AI as cheating, but a generation of students, AI, you know, they've grown up with AI in the same way that the internet natives have grown up with the network. If we continue asking them these traditional questions about, like, find a palindrome in this string or measure the volume of rainfall in this graph, right?

Like, are they going to learn anything from a traditional CS degree? The answer is obviously no. So, like, if we know that we cannot stay here and there's plenty of things that need fixing, like... Let's burn it down and do something that matters and figure out how to use this appropriately. And I think finding ways to assume that they're using AI and what are the new learning objectives and how do you make that still frictionful in the ways that cause them to think and learn about different things, like almost certainly at a higher level. Undergraduate education and curriculum design and assessment.

Well, it sounds like whoever's, so the professors, the The TAs, the people who are designing this course, they also need to put in a bunch of work because no one has this figured out. You need to experiment. You need to first use the tools yourself. You need to come up with novel ways of creating an assignment which you need to use AI to solve, but AI itself is not enough. How do you get the feedback? You might need to put in more time, let's say, sit down with someone afterwards and have them experiment. And one of the interviews that seem to be working well is when you're pairing with someone, like video or in person, you're using AI, but you're explaining what you're doing and how, right?

You cannot just fully fake it and go away, give it to a friend or just have Claude do it. You're now showing your cognitive ability. The interviewer can push you into uncomfortable, you know, they'll get a gauge of like how expert you are and push you further and further and so on. Yeah, exactly. And at the undergraduate level, like there is some move towards more oral exams and those sort of things.

And as closing, for...

For grads right now, college students sitting inside of CS programs, software engineering programs who are seeing, okay, it's more competitive to get into the industry, tools are changing. What advice would you get in terms of tactics, strategies to activities to do to maximize your chances of getting into the industry and then doing well there?

This might be risky, but most of my success in life has come from not doing the thing that everyone else is doing. Like it is really incredibly hard to stand out if you are just listening to what is a hot topic and chasing the same shiny thing that literally everyone else is doing. Right. And so I think it is to thine own self be true. Right. Like find something that you are actually interested in and put in the time doing something that you care about, especially if it's weirder than what everyone else around you is doing, because then you will have a novel story to tell your interviewer. You will be worth talking to at a cocktail party so that the startup founder is like, yeah, I want to take a risk on this kid, right?

And the more time that you put into these things, the more expertise that you can bring to it, the better off you're going to be. I have a worry that with AI-assisted engineering, Our emphasis on raw engineering skill is dropping. Like you can get by with a lot less of that depth. But the only way to replace that is you need to understand the problem domain that you're operating in to a much deeper level. And I think that that might actually lead to less mobility in the industry. Like it's not, I'm a C++ engineer or I'm a backend engineer. It's a, I'm an engineer in embedded audio. Right. Like you understand those problem domains to a much deeper level and that's your actual advantage.

As closing, what's a book that you would recommend and why outside of your excellent book, of course?

I mentioned it earlier, but I'll stick with it. Psychology of Software Teams by Cat Hicks, especially like there are a couple of chapters in that that I could have that downloaded into the brains of every leader around. I would be so happy. Like the brains in jars analogy is. I'm glad I got to finally sit down with Titus, and this conversation did not disappoint. To start with, I knew that Google's codebase was large, but I had no idea we're talking about 250 million lines of just C++ code. And this was all pre-AI, so we're not talking about over They were both code bases generated by AI. As comparison, Uber's mobile apps had code bases of about one to two million lines of iOS and Android code back in 2016, and Uber crammed a lot of business logic into those apps.

How Titus and the team cleaned up these foundations Without using AI was also pretty educational. They built deterministic code generation tools to parse the code, match patterns, then generate edits. This is an approach that still could work more reliably even today than just asking non-deterministic AI to refactor a codebase, even though it is arguably more difficult to write advanced parsing logic on a codebase like Titus and team did. I also appreciated Titus' view on where AI is helpful today. He mentioned this quote he heard from a senior engineer that said, we are up 70% on the amount of code, but we're also up 30% on outages.

The number of features is the same. Well, that doesn't sound all that good. Where's the value from AI then? Titus says that we won't get all that much value from doing the same stuff we did before AI, but faster. The value is to be more ambitious and do harder things that we would have never attempted before AI. Finally, I hear a lot of worry for new grads in tech about how they'll get a job or not, especially when AI agents are so commonly used, but CS programs have not incorporated them into their program. Titus is one of the few engineers who has been part of the steering committee for the CS curriculum and says that this program needs to either change substantially and be more practice oriented or CS enrollment will plummet.

Titus also says that it might make sense to finally have a software engineering curriculum that is focused on students getting far more experience building software with these modern tools. I don't think any of us has an answer to what will work, but having more people from the industry get more involved in refreshing the CS software engineer curriculum sounds like a step in the right direction to me. Check out the show notes for related to Pragmatic Engineer deep dives on Google's engineer culture, measuring developer productivity, and the impact of AI and software engineering. If you enjoyed this podcast, I'd greatly appreciate it if you left a rating on it and perhaps a review.