# Better evaluations can expose gaps in representation

Markdown copy of https://realalpha.app/s/better-evaluations-can-expose-gaps-in-representation/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Tulsee Doshi](https://realalpha.app/builders/tulsee-doshi/) · Tulsee Doshi is head of product for Google’s frontier AI models. · Leads the Gemini product roadmap and strategy after five years heading Google's Responsible AI product work.
- Host: [Lenny Rachitsky](https://realalpha.app/people/lenny-rachitsky/) · Host, Lenny's Podcast · Spent seven years at Airbnb; his product newsletter has over 1M subscribers.
- Show: Lenny's Podcast
- Episode: [Why Google wants to win the frontier conversation | Tulsee Doshi (Head of Product for Gemini)](https://realalpha.app/v/qGf2Yvan4Wg/), Nov 8, 2026
- Moment: 26:30 to 29:51 of the episode (3 min), playing on https://realalpha.app/s/better-evaluations-can-expose-gaps-in-representation/; the episode from there: https://realalpha.app/v/qGf2Yvan4Wg/?t=1590

## Quote

> “Realistically, six skin tones **cannot capture the diversity** of skin tones in the world.”
> — [Tulsee Doshi](https://realalpha.app/builders/tulsee-doshi/) at [27:19](https://realalpha.app/v/qGf2Yvan4Wg/?t=1639) · Lenny's Podcast

## Takeaways

1. The Fitzpatrick scale was too narrow for global representation.
2. Use evaluation scales suited to the problem being measured.
3. Test the full stack users experience, not just the model.

## About this segment

Tulsee Doshi explains why Google moved beyond the six-tone Fitzpatrick scale to a 10-tone scale for evaluating products and models. She also stresses testing the full experience, including serving and judging.

From “Why Google wants to win the frontier conversation | Tulsee Doshi (Head of Product for Gemini)”, Nov 8, 2026. [Watch the full episode](https://realalpha.app/v/qGf2Yvan4Wg/)

## Transcript

From the transcript Lenny's Podcast publishes with the episode.

[26:34](https://realalpha.app/v/qGf2Yvan4Wg/?t=1594) Oh, oh boy. So this is Eli Collins, VP product. He's a colleague of yours at Google DeepMind. Okay. So one is you worked on something called the Fitzpatrick scale and the monk skin tone. And this was a big shift in how people in how models worked around and evals were a core part of this. Share this story. What this work was. Yeah. So I like like you noted earlier,

[26:58](https://realalpha.app/v/qGf2Yvan4Wg/?t=1618) I spent six years in responsible AI and a big part of that time we were looking at skin tone and skin tone representation in our models, in our products. And one of the things we realized is that early on there was a scale that individuals were using the Fitzpatrick scale, which was six skin tones. And that was how we were evaluating the effectiveness of product. And Realistically, six skin tones cannot capture the diversity of skin tones in the world.

[27:27](https://realalpha.app/v/qGf2Yvan4Wg/?t=1647) Not only that, it was actually built for an entirely different purpose. It was built for like health care purposes. And so it actually skewed extremely light skin tone, given the problems that it was trying to solve. And so it actually not only was only six skin tones, it was also not even the six like correct skin tones if you're trying to represent like a global population. And so we worked with a professor at Harvard, Professor Monk.

[27:50](https://realalpha.app/v/qGf2Yvan4Wg/?t=1670) He had been from like a more anthropology, sociology perspective, looking at characterizing a set of skin tones. And so he had a scale of 10 skin tones that he had built through a number of user interviews and interviews. Research that he had been doing over years. And we pulled that set of skin tones into our evaluations and actually redid a number of the ways that we evaluated our products and our models to reflect that 10 skin tone scale and actually use that to improve things like our efforts in the pixel camera, like some of our image generation to be more intentional about how skin tones are represented.

[28:26](https://realalpha.app/v/qGf2Yvan4Wg/?t=1706) That is awesome. That is some serious impact you had here. What else do you think is maybe useful for people to hear as they hear this and they maybe have to rethink their evals that may be out of date?

[28:37](https://realalpha.app/v/qGf2Yvan4Wg/?t=1717) Yeah, I mean, I think one thing that's kind of interesting about evals in general, especially now, is you have to kind of think about them holistically, right? So one of the things that we do, for example, is you Train. You have judges that evaluate the models and how well they perform. The quality of the judge also matters. Right. So your model needs to be performing well, but also how your judge rates the model also needs to be performing well for you to have an accurate sense. Right.

[29:04](https://realalpha.app/v/qGf2Yvan4Wg/?t=1744) Whether you evaluate on your production stack with all of your mitigations and filters might affect your model performance. Right. How your model is served might impact your model performance. And so all of these pieces need to be thought through as you're trying to evaluate what an end user will actually experience. And like this is something that I talk about a lot with our audio team, actually, because audio is complex.

[29:26](https://realalpha.app/v/qGf2Yvan4Wg/?t=1766) You're talking not just about. The actual content that the model is saying, but also the naturalness of the voice and the tenor of the voice and its inflections. And these are things that serving actually can really affect, even if your model is good. And so having a clean setup that you're actually evaluating end to end becomes super, super important.