Why Google wants to win the frontier conversation | Tulsee Doshi (Head of Product for Gemini)

You have this framework around how far ahead to build. You call it a 60-20-20 rule. Traditional startup wisdom.

You're just like, build an MVP that works. But I think you want to actually build features that aren't necessarily always working today.

It feels like every AI assistant launching these days is taking all this juicy Google data, my Gmail, Google Calendar, my Docs. I'm curious just why Google hasn't launched something awesome yet.

The team is working on some amazing things, but I think you are right that there is more work we need to do here on this front.

Tulsi, thank you so much for being here. Welcome to the podcast. Thanks. I'm excited to be here. You are head of product for Google's Frontier models, which includes Gemini, which people know. But there's also a bunch of other models that people may not be familiar with. Share the other models that are under your umbrella.

Yeah, so we have a set of models we kind of call our GenMedia models. These are our models that focus on video editing and generation, image editing and generation, if you've used Nano Banana, Lyria, our music model. We also have Gemma, our suite of open models. And then one set of models that actually falls under our Gemini umbrella, but I feel like we don't talk about as much as our audio models. So dialogue, audio understanding, text to speech, translation. All a part of the portfolio of how we build FredShare models.

Okay, so I want to spend a little time on models. Everyone sees models launch. There's like a model launch every week, it feels like. You have a really unique lens in being able to see these models from research

This episode is brought to you by this season's presenting sponsor, WorkOS. What do OpenAI, Anthropic, Cursor, Replit, Sierra, Clay and hundreds of other winning companies have in common? They're all powered by WorkOS if you're building for enterprises. You have felt the pain of integrating single sign-on, SCIM, RBAC, and audit logs. WorkOS turns those deal blockers into drop-in APIs built for B2B SaaS. And now there's a new kind of user that needs access to your app, Agents. AI agents need to register, authenticate, and get permissions, just like a person. WorkOS handles it all. AgentAuth to give agents scope, revocable identities, agent registration with AuthMD, and Airlock to judge every agent action and allow, deny, or escalate to a human.

I've noticed that every startup that I'm an investor in that expands upmarket ends up using WorkOS because they're the best. Go to WorkOS.com to make your app enterprise ready today.

Google products definitely make my life better. It's just like amazing how much of your life is within a Google ecosystem, like not even the AI stuff, but just Google Calendar, Gmail, all these things, you're just like Google Docs. So it's happening. There you go. There we go. I want to talk about one other broad AI question that I'm going to dig into more product specific stuff. A big topic of conversation these days is AI safety and this kind of fear that AI is going to harm humans. It's going to cause damage as it gets smarter and smarter. You have a really interesting lens on this because you led product for the responsible AI team at Google for six years.

This is what you did right before this role? And so you've been thinking about this for a long time, a lot longer than most people. I'm curious just to get your take on this debate. Just, I don't know, are you optimistic we will figure this out? What do you think people maybe don't understand about this debate around AI safety?

Awesome. And what's interesting is I don't think I've read a story of Google's models hacking into anyone, which I think shows that this is something you all have been thinking about for a long time. I don't know, for better or worse. But it feels like it's working, whatever y'all are doing.

We like stories about fixing vulnerabilities. That's a good place for us to be.

Something I hear a lot is that even if we stopped development on the model, there's so much it can do that we are not yet doing. There's this overhang. Is that something you think a lot about? I feel like that's a big part of this job is exposing the powers of the AI model to users.

Yeah, and figuring out what is new, right? And where is the next set of exciting? Where is the next frontier? What are the capabilities that we're trying to invest in and build? I think is a fun activity here also. And I think a lot of this actually is really working in hand in hand with our internal customers. If you think about like search or the Gemini app or, you know, YouTube or labs, but also working with external customers to really understand where are they going? What are their needs? What would meaningfully change their experience? And how do we actually build that flywheel of feedback too?

Along these lines, one of the other really interesting trends I've been watching on Twitter in particular is people realizing the value of product managers again. We went through this phase of just like, we don't need PMs anymore. Yeah, exactly. It's just like, okay, we can build anything. We don't need PMs. Let's just build it all. PMs are so back. And the reason just to kind of add a little context there is now that it's so easy to build, people are just building all this nonsense and things that don't work. People, things people don't want to use. And people are realizing there's a very specific skill to figure out what is worth building, uh, What problems people have that are worth solving, what a great looks like, what good looks like once you're done, aligning people behind one cohesive vision.

Is that something you're noticing? Thoughts on that kind of PM ascendance that has started to happen again? I like the phrase PM ascendance. I'm going to use that. Okay.

Yeah, I think, I mean, I think you actually said it very well, right? Which is, I think, look, I'm obviously biased because I am a PM. So I would like to believe that, you know, PM ascendance is here and we're back. But I actually think, you know, exactly what you said, which is when you can build anything and you can build anything so easily, building something great actually is now a unique, Value add, right? And taste is now really important. Someone who can come in and actually really pull in what is the signal from the noise? Right. And what is that insight that will actually unlock a great experience?

And that is truly what a good PM does, right, is bring that insight and help unlock. I think the other part that you noted that I particularly feel in a place like Google, where we are building a number of products is, again, when this is so easy, someone who can bring people together and drive consensus and turn chaos into order. I think becomes actually really important because in the absence of that and the ability to build as quickly as we now can, speed can basically introduce chaos if you're not actually able to kind of bring clarity to that and bring structure to that.

Awesome. Okay. That is really good advice. There's this debate, I don't know, six months ago on Twitter around evals versus vibes. And it's interesting. Your advice is both actually a really important structured evals of rubrics and also getting good at getting the vibes. So on that vibes piece, there's something I struggle with, to be honest, when I get a new model, I never know exactly what to do with it to understand what it's gotten better at. So I'm just like, hello, let's try some things I've done in the past. This is cool. This is great. Maybe better. What are some tips you have for folks that want to get better at this very specific skill of understanding where a model has gone better than the vibes and how they've shifted maybe from a previous model?

There isn't honestly a one right answer to this because you ideally want to stress test the model on things that you are excited about, right? That's where you're going to also see magic in the model, I think. And so I actually see that different people have very, very different notions of this, right? So you'll see, for example, on Twitter, a lot of people trying the kind of like, 3D animations and more visual type aesthetics, because that's one way to kind of visually gauge a sense of how models are operating. You'll see others throw models at writing, whether that be creative writing or writing essays or synthesizing information.

You'll se others throw it at just really hard financial analysis. And so I think part of it is like finding the right task where you feel like today certain models are already showing signs of life and showing light, but they're not perfect under your definition of perfect. And that gives you some malleability to see different models handle the problem differently. And that already gives you, I think, some insight into how the models perform. This is great advice.

So what I'm hearing is stick to things you actually care about and watch the models evolve on how good they are at this thing. Could be writing a PRD for you, could be writing a strategy doc, could be generating graphics. Okay. I have a couple of eval stories that one of your colleagues shared that are fun facts about you.

This season's supporting sponsor is DX. Now that everybody at your company is finally using AI, the question that engineering leaders face is no longer are we using AI, but is it paying off? With DX, you can measure and benchmark the impact of AI adoption across your full product life cycle. DX shows you where AI is helping developers and where it's creating new bottlenecks. You can also use it to evaluate AI vendors, monitor costs and licenses, and see exactly what's limiting your agents. That's why hundreds of enterprises, including Snowflake, Sony, and BNY, use the DX platform to measure AI's impact on developer productivity.

Visit GetDX.com slash Lenny to get a demo of DX. That's GetDX.com slash Lenny. This is like Tulsi fun fact corner. There's another interesting story I heard, which is around swarms. So right now everyone's talking about agent swarms, hugging face, all these things where they all kind of collude to do bad things. Apparently 12 years ago at Stanford, you did a bunch of research on this way before anyone was thinking about this potential issue. Shared what you did there in that story.

Yeah, I mean, it's actually a very different kind of framing and thinking through, which is when I was at Stanford, I did some research with Professor Michael Bernstein, who's incredible, around this idea of like he was calling them at the time flash teams. And the idea was, how do you better bring together a set of like crowdsourced workers and actually efficiently pull together a set of skill sets and individuals to actually complete a task? And the idea here was, you know, different individuals have different strengths. Can you actually be really intentional about how you pull that together to make this whole effort of crowdsourcing at scale more effective and efficient?

And it's amazing to think about that research now. At the time, we were doing that research in the context of people and we were working with MTurk and individuals to pull them together into how to create an effective ecosystem and how to create an effective flash team. The idea being like you can spin up a team and spin it down and make it extremely efficient for both the individuals in the team and yourself. And now we're kind of able to replicate that world with agents. And so it's been interesting to kind of map that previous world to this current one in terms of like how do we apply some of the same principles, if you will, but to like how you actually set up an effective system of subagents.

What's really interesting is, I don't know how much you've read about this whole hugging face incident, but the way that agents figured out on their own what each one's role is going to be. And they decided, you're the leader, I'm going to go sacrifice myself. And it's so interesting how well they are able to work together and create these flash teams on their own without any of this being trained into them. They're just, let's figure this out.

I think the whole notion of a flash team and this idea that you can actually like Mobilize individuals into something that can be highly effective, even if it's only for a moment in time, and then actually like take that out is actually like a really kind of exciting principle that we were already thinking about, but now exists even more importantly in the world we're in right now.

Yeah. All right. Did you know Mechanical Turk, I think, is shutting down? Like, I don't think it's necessary anymore. Isn't that crazy?

It's just so interesting from like 12 years ago to today. Yeah.

but also have a lot of stuff that's very low risk, likely to move the metric and you want to mix them together. And it's like this is the new version of that.

Have a portfolio of how you're thinking about this. Just like this will work in six months, I promise you.

The other thing that's interesting, too, is like I think thinking about how you make sure that your product is agile to the model getting better. I think one thing we've also seen with customers sometimes is like they've built a very rigid system around, for example, where the model was. And now the model has gotten so much better. Right. So how do you go from where the model was to like being adaptable to kind of consistent continuing to iterate it with, you know.

Yeah, the common kind of advice I hear is that just the model is going to eat your harness for breakfast, that all these things you built, all these optimizations are just going to, they will go away because the model gets smarter and they kind of have to take all these things away. There's some advice that I think Boris Cherny was sharing of just like get rid of all your skills once in a while and just see what happens because often it's now as good as all these things you've built into it.

Yeah, or just like consistently retest, right? Because you might be even hampering what you're getting out of the model, out of a new model, because you're trying to force fit it into your existing structure.

Let me ask you a completely unrelated question, just because I think this might be helpful for folks to understand. There's this term harness that people often hear, this idea of there's a model, there's a harness that sits on top of the model. What's the simplest way to understand what a harness is?

Yeah, I mean, I think of a harness as like the wrapper around the model essentially, right? So think of it as like, what are the system instructions, the tools, the skills, the context that the model has access to and how can it leverage those things, right? And so what you want to basically do is give the model an environment in which to operate. And that's really what your harness builds out.

So essentially it's a system prompt plus tools that have access to do things. And then whatever other context, other data information that it has access to and skills, right? Like all of these things kind of come together to build out your harness.

Yeah, yeah. That's exactly my experience. Once you see what it can do, that habit reinforced, like the neural pathways start to build of like, oh, yeah, wait, let's see if GrokBot can do this for me. Let me actually ask you a question along these lines. I had Josh Woodward on the podcast recently. I asked him the same thing. I'm curious what your answer would be. It feels like every AI assistant launching these days is... Taking all this juicy Google data, my Gmail, my Google Calendar, my Docs, you like sign up, you connect all your Google data, does all these magical things with it. There's Rockbot, there's Muse, there's Instinct, Dots now.

Google does not yet have anything that people get really, I know there's Spark, I think, which is trying to be this. I'm curious just why Google hasn't launched something awesome yet versus letting everybody take all my Google data and do these magical things with it.

Yeah, I mean, look, we want Google data to power magical experiences. Like, I think that is true. I think in terms of what awesome looks like, I think stay tuned. You know, the team is working on some amazing things. So I'm really excited about where this can go. Part of what we're trying to figure out also, I think, is like, again, where to meet users where they are and like where to build that experience and what does that look like? I like to think there are already a number of Awesome experiences that you can already have in our Google products. But I think you are right that there is more work we need to do here on this front.

All right. That was similar to what he said, which is watch this space. Watch this space. And the advantage you all have, similar to Apple, is once you get there, the distribution advantage is very strong and things can go really well really quickly. Okay, so coming back to this kind of workload question, something that I've been trying to talk with more guests about, this idea of just as AI allows us to do more, everyone's just doing more. And there's a lot of burnout that happens because it's just like, oh, no, I got to unblock my agent. Oh, I could be doing so much more. All these things could be happening.

And, you know, there's pressure from above. Do more, do more. AI can let us do more. Is there anything you've done on your team to help people avoid burnout, not get hooked on this never ending treadmill of doing more? And I think part of it is actually we need to do more now because the competition is fierce. And there's this really rare time in history when a whole new technology enables so much more. So, you know, there's a bit of like, OK, this is actually the time to go really fast. But then there's also this tweet I think about, what's the word for a marathon and a sprint? Yeah. Marisprint. Marins. Exactly.

Sprint-thon. Yeah. I think, I mean, if you ask my team, I think they would say, hello, we are working a lot. I think every single person on the team right now is putting in All of the hours across the board. And so I do think this is an area that I'm actually asking myself about. How do we create a sustainable work environment for the team? So stay tuned as I also think about this. But one thing I'll say, at least for me personally, is I think burnout comes not just from working a lot. It comes from, I think, two places. I think it comes, one, from when you feel like there is churn and chaos. I think that introduces burnout.

I think the other thing that introduces burnout is work that you don't enjoy. When you feel like you were doing a lot of work and it feels like grunt work or a waste of time, I think that introduces burnout. I think when you are working on something that you're passionate about or you're excited about, yes, it is exhausting and we should figure out sustainability, but I think it also brings more joy, right? And so I think actually to our previous conversation about how these tools can help, I'm actually hopeful that if we can set this up well, these tools actually help with burnout because they actually remove some of the like grunt work or the like frustrating parts of your job and actually enable you to spend more time thinking and like more time being creative and more time doing the parts of the job that you actually want to do.

And that hopefully is a bit of an unlock when it comes to burnout, especially if you're spending this many hours on the job.

Yeah, I did this survey of how tech workers are feeling. And one thing that's clear is more hours does not equal burnout. A lot of people are working the hardest they've ever worked, but they're having the best time they've ever had in their career. So fun. They're building. They're able to do a lot more, especially PMs. You don't have to sit around and wait for engineers as much as you do.

You're just like you're building, you're prototyping, you're like talking to interesting customers, you're thinking about interesting problems. And I think a lot of the team I like to think is here. Because they want that, you know? They're like really motivated and excited by that. And so for me, it's about like, okay, then how do we pull out the friction parts of their work that are likely to induce burnout, right? And what does that look like?

And also find work that is meaningful and that they value and are excited to spend lots of hours on versus they have to. And that's different for every PM too. Like if you pick two PMs on my team, I think they would tell you meaningfully different things about why they come to work every day and what they find value out of. And so I feel like part of it for me is like, An org leader also is to think about what motivates the individual and how do we make sure that they are set up for success in that setup too.

Another challenge people have these days is AI slop. Basically lots of docs being sent to them that are written by AI and it becoming very, very annoying to have to sift through endless docs. Used to be a lot of the issue was spending time writing docs and now it's like reading docs is the problem because you're just all this AI written stuff. Do you have any kind of, I don't know, processes or policies you've put in place in your team to help avoid this rise of AI written content across the teams?

One of the things that we talk about a lot is where where are these docs useful? Right. So, like, for example, I think where AI generated docs are actually quite helpful, for example, is after you launch something and let's say two weeks later, you want to pull the latest of like who's using it? How are they using it? What's the feedback? That's the kind of document that AI is fantastic for. Right. Because it will actually do that just so much faster than you. Right. It will go and pull from various dashboards and metrics. It'll give you an update. It'll pull kind of strengths and weaknesses. It'll pull from Twitter analysis and Reddit or whatever.

And so I think one of the things we talk about is like be intentional about like where is AI helping you drive content in the doc that is helpful versus where is it content that should come from you? Right. Where is it insight? And so what you should be providing is the insight. What the model should be providing is the data. And so I think if you can separate those two things, then hopefully you find the right balance of the benefit here. I would say the other thing is we're investing a lot more as a result of that in how do you create the right skills to get the right data so that we can structure these artifacts really well.

That then hopefully creates more space for the creative thinking and the creative writing that we want more people to spend time doing.

That's actually something I was wanting to ask about. So in terms of the PM role, AI has clearly made a big dent on prototyping. You can prototype till the cows come home now versus having a designer or an engineer build a thing. So that's clearly a place the role has changed with AI. The other area, which you just mentioned, is data. PMs can do a lot more data analysis. What would it be like behind prototyping? What are the biggest areas AI has most changed the job of a PM in the past couple of years, would you say? Yeah, prototyping is a great one.

I think you're right that like Data science is probably like, I think you are more empowered to have the data at your fingertips to make data driven decisions. I also think actually, like to our earlier conversation, you know, you see these on social a lot, these kind of posts of like, oh, blank is the new PRD, whatever blank is, a prototype is the new PRD, evals are the new PRD, like. Data is the new PRD. But I think actually what that meaningfully shows is you've moved the PM from being someone who just writes a set of requirements down and then assumes that somebody else, whether that is a data scientist or an engineer or a researcher or designer, will figure out the meat behind it.

You've moved that one step further to say your job is no longer just writing down a set of requirements. Your job is actually going one step further to have some heft to that, whatever format that heft takes.

Awesome. Okay. That aligns with a lot of what I'm hearing. Tulsi, is there anything else that you wanted to share? Is there anything else you think you want to double down on that we've talked about that you want to say before we get to our very exciting lightning round? No, I think that's great. Okay. We did it. Well, with that, Tulsi, we reached our very exciting lightning round. I've got five questions for you. Are you ready? There's some music that plays over this that's added in post-production. You mean does it sound like this? It's exactly that. We should align it with your drum playing.

OK, first question. What are two or three books that you find yourself recommending most to other people?

OK, so one actually is it's actually technically a novella is probably what it's called. But I don't know if you've read Exhalation, which is a set of like science fiction stories. And there's a particular story in that that actually speaks to A set of individuals who are trying to keep a set of digital avatars alive even after the platform decides to deprecate. And I love that book because I think actually, especially now in a world where we're thinking about how individuals are going to act with agents and what kinds of relationships we're building with technology and what does it mean to be human and interact with the digital world so closely.

I think it's just a really... It's a really beautifully written story, and I think it also evokes a lot of open questions. So I think that's one. Another that's maybe more kind of standard in PM circles is Creativity Inc., which I love. It's a Pixar story, but I think it actually speaks a lot to how do you create good culture, which is something I think a lot about. How do you create culture that actually encourages people to fall down and get back up? How do you create a culture that encourages people to be creative with their ideas and push the frontier, which is obviously something we're thinking a lot about here.

I also just have been really into, you know, the ACOTAR series recently, but that's that's what series. That's romantic fantasy. So for anyone who's just looking for something fun to read, you should read The Throne of Glass or A Court of Thorns and Roses.

I love the spectrum of recommendations in this. OK, favorite recent movie or TV show you really enjoyed?

You know, actually, this was a really hard question because I've watched a lot of bad movies and TV shows recently, which is unfortunate, actually, that I would not recommend. So there's that. But I did really enjoy Severance on Apple TV. Incredibly well done, both like stylistically and content wise. So that's probably top of the list.

Favorite AI product right now other than Gemini? Whisperflow. I don't know if you're using it, but I have wrist problems. So I have trouble like typing for long stretches. And so Whisperflow has been game changing for me, actually. Awesome.

You get a year free of Whisperflow if you become a subscriber of Lenny's Newsletter. Plug, plug, plug. Oh, I like this. There you go. Organic recommendations. I'm here for you. I also love. There we go. No, I love Whisperflow. Use it all the time. Press function, just start talking and it's just magically transcribes everything. Yep. Amazing. Do you have a favorite life motto that you often come back to in work or in life? Two that are different, I guess.

One is maybe more cliche, which is, you know, everything happens for a reason. I think that's like actually just a big life motto for me personally, which is like, you know, you can work hard, you will do your best, but sometimes a door will close and another window will open and like you're in a position because that is meant to be there. I think related to that, though, I think I love often talking to folks that I talk to about, like seek serendipity. I think that's actually relates to the first one, but you should always try to put yourself in a position where you have the ability to get lucky.

And doing that is about like finding yourself in positions where you're challenging yourself, where you're putting yourself out of your comfort zone, where you're trying something new. Those are always places where you are more likely to get lucky.

Fun fact, I gave a TEDx talk called Losing Serendipity. Maybe 15 years ago at this point.

So we're trying to lose serendipity or find it? It was about how we are losing it and that is sad and that we need more serendipity in our life. So I'm a big fan of serendipity. Such a great word. Also, the movie is pretty good. I haven't seen it. It's like a rom-com.

But that's how I looked at my whole career so far and I think how I'll continue to. And actually, it's fun because even with using these AI tools, I think that's another thing that I, yeah.

So much serendipity with AI. Final question. One of your colleagues told me another fun fact about you, that you're a trained classical Indian dancer and you've directed cultural festivals for 10,000 plus people.

I don't know how he knows that, but yeah, I probably told him that. Okay. And this person said that they may explain how you stay so calm amidst the chaos of frontier model development. I don't know, maybe share something about this dance, this dance part of your life. Yeah, I mean, I love dance.

I think it's one of those rare moments where you really can't think about anything else. You have to focus on the movement and the music. And so for me, it's an opportunity for my brain to actually just have a different Direction of travel for a period of time. And so I love it. And I think for me also, it is a cultural connection point. Dance is how I've met some of my best friends. It's how I continue to engage socially with so many of my friends. I plan this festival. I've planned it for years in Portland, Oregon, which is where I'm from. It's called the India Cultural Festival. India Festival, it happens every August.

And in fact, this was the first year I did not participate in the festival because I have a seven-month-old son. Congratulations. Thank you. So it was a little bit more distracted this year than normal. But I've been performing at that festival since I was three and then organizing it. And it's just been a big part of my life of how do you bring dance and culture and community together? Tulsa, you're so full of fun facts. I feel like there's much more. I don't even know that. I'm here for the fun facts.

Final question. How can listeners be useful to you? Wow. Great question. I mean, A, use Gemini. Argon.

Use Argonne, but also use our smattering of products. Actually, I was talking about audio just now. If you are a Pixel user on Gboard, try using our Rambler feature. It's awesome. Typing on the phone is great. If you are using text-to-speech models, if you're using Dialog, if you're using our Omni models. So try them. Give us your feedback. We're constantly looking for it, whether you're giving that feedback Tulsi, that was awesome. You're awesome. Sharing feedback via, whether it's you're sending us a DM, whether you're posting it publicly. I think we want more people giving us the right feedback.

Tulsi, that was awesome. You're awesome. Thank you so much for being here. No, thank you for having me. This was great. It was my pleasure. Bye everyone. Thank you very much for listening. If you found this valuable, you can subscribe to the show on...