# Research PMs should test models and start with small evals

Markdown copy of https://realalpha.app/s/research-pms-should-test-models-and-start-with-small-evals/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Tulsee Doshi](https://realalpha.app/builders/tulsee-doshi/) · Tulsee Doshi is head of product for Google’s frontier AI models. · Leads the Gemini product roadmap and strategy after five years heading Google's Responsible AI product work.
- Host: [Lenny Rachitsky](https://realalpha.app/people/lenny-rachitsky/) · Host, Lenny's Podcast · Spent seven years at Airbnb; his product newsletter has over 1M subscribers.
- Show: Lenny's Podcast
- Episode: [Why Google wants to win the frontier conversation | Tulsee Doshi (Head of Product for Gemini)](https://realalpha.app/v/qGf2Yvan4Wg/), Nov 8, 2026
- Moment: 21:22 to 24:39 of the episode (3 min), playing on https://realalpha.app/s/research-pms-should-test-models-and-start-with-small-evals/; the episode from there: https://realalpha.app/v/qGf2Yvan4Wg/?t=1282

## Quote

> “**start with even just five prompts**, right? Like start, you don't have to start large.”
> — [Tulsee Doshi](https://realalpha.app/builders/tulsee-doshi/) at [23:29](https://realalpha.app/v/qGf2Yvan4Wg/?t=1409) · Lenny's Podcast

## Takeaways

1. Hire PMs who use models and stress-test them personally.
2. Start with five prompts; even 20 can reveal model differences.
3. Define good examples yourself before scaling an evaluation.

## About this segment

She says research PMs need model intuition, hands-on testing and adaptability, then recommends starting evaluations with a few prompts. Judged examples can reveal differences before teams scale a benchmark.

From “Why Google wants to win the frontier conversation | Tulsee Doshi (Head of Product for Gemini)”, Nov 8, 2026. [Watch the full episode](https://realalpha.app/v/qGf2Yvan4Wg/)

## Transcript

From the transcript Lenny's Podcast publishes with the episode.

[21:25](https://realalpha.app/v/qGf2Yvan4Wg/?t=1285) So on that, as you're hiring PMs these days, is there anything you... Look for more and more kind of a skill that might be trending up in the PMs that you hire that you think is really important to be really successful or just broadly skills that you find are really important to be successful in this role of being a research PM.

[21:44](https://realalpha.app/v/qGf2Yvan4Wg/?t=1304) Yeah, I mean, I think more and more so we are looking for PMs who, you For lack of a better way of putting it, understand the vibes of our models or other models. I want PMs who have an opinion, who are playing with these models every day, who have their own use cases for how they judge whether a model is really good, who play with and stress test these capabilities themselves, who are capable of diving in and looking at data and getting scrappy and rolling up their sleeves.

[22:17](https://realalpha.app/v/qGf2Yvan4Wg/?t=1337) And so I think it's both PMs who are trying to build that intuition and As well as who are technical enough in the sense of being willing to kind of like, like I said, roll up sleeves, go in, really understand the data, build an eval, right? Actually sit down and pull those pieces together. I think that's really important. The other thing I'll say is I think PMs who are agile and adaptive are like also really important.

[22:44](https://realalpha.app/v/qGf2Yvan4Wg/?t=1364) I think the space is changing every two weeks. A roadmap you wrote two months ago. You're going to be looking at it again and rewriting it. Right. And so I think PMs who are constantly keeping an eye out on the environment, on the market landscape and actually consistently reinterrogating, like, what are we doing and why are we doing it? As opposed to assuming that there was a fixed kind of plan of action.

[23:05](https://realalpha.app/v/qGf2Yvan4Wg/?t=1385) This is awesome. OK, so vibes, evals, adaptive. I want to poke at these a little bit. So with evals, say someone doesn't really know how to write an eval, how to approach writing an eval, their first eval. What are some tips to start getting better at the skill writing evals? Yeah.

[23:20](https://realalpha.app/v/qGf2Yvan4Wg/?t=1400) And to be honest, we're still figuring that out internally with our PMs as well. And I would say like we see some PMs who are stronger at this than others. And so we're kind of working to upskill our whole team here. I think for me, a big thing is start with even just five prompts, right? Like start, you don't have to start large. I think we often think about like, oh, I need to build some sort of massive, large scale evaluation or benchmark.

[23:45](https://realalpha.app/v/qGf2Yvan4Wg/?t=1425) I think often like even just a mini evaluation where you get to like 20 prompts that can really stress test a set of capabilities where you can see signal between your model and another model or model A and model B. I think that already is so powerful. And that's an unlock because it doesn't require you to think as critically about all of the pieces. Right. Like, start simple. Start with even just yourself, like you as the Raider, you looking at a prompt, you trying to decide if the results in the response is good, especially when you're talking about these like long, multi-step, multi-turn trajectories.

[24:21](https://realalpha.app/v/qGf2Yvan4Wg/?t=1461) Just even being able to identify how you're thinking about good is really hard, actually. And so even getting that set of good examples will already give you a sense of, oh, this is what I'm trying to get to. It'll give yourself clarity. And then you figure out how to scale that up, I think.