# Qualitative tasks need broad reward criteria

Markdown copy of https://realalpha.app/s/qualitative-tasks-need-broad-reward-criteria/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Jack Rae](https://realalpha.app/builders/jack-rae/) · Research Director, DeepMind · Led Gemini Thinking, co-led Gemini pre-training, and contributed to Gopher and Chinchilla language-model scaling research.
- Host: [Jacob Effron](https://realalpha.app/people/jacob-effron/) · Partner, Redpoint · Growth investor in Abridge, Ramp and Physical Intelligence; ex-Flatiron Health.
- Show: Unsupervised Learning
- Episode: [Noam Shazeer and Jack Rae: Scaling Test-time Compute, Reactions to Ilya & AGI](https://realalpha.app/v/atMRWzgHEGg/), Mar 17, 2025
- Moment: 9:39 to 11:58 of the episode (2 min), playing on https://realalpha.app/s/qualitative-tasks-need-broad-reward-criteria/; the episode from there: https://realalpha.app/v/atMRWzgHEGg/?t=579

## Quote

> “take in a very broad set of criteria and then **apply a reward signal** once we have the reward signal we can train with reinforcement learning against it”
> — [Jack Rae](https://realalpha.app/builders/jack-rae/) at [10:54](https://realalpha.app/v/atMRWzgHEGg/?t=654) · Unsupervised Learning

## Takeaways

1. Rubrics can describe qualities such as style and interest.
2. Broad criteria can produce a reward signal for training.
3. This extends reinforcement learning beyond easily verified tasks.

## About this segment

Jack Rae says models can be trained against broad rubrics for qualities like style and interest. That offers a path for reinforcement learning beyond tasks with automatic right answers.

From “Noam Shazeer and Jack Rae: Scaling Test-time Compute, Reactions to Ilya & AGI”, Mar 17, 2025. [Watch the full episode](https://realalpha.app/v/atMRWzgHEGg/)

## Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[9:43](https://realalpha.app/v/atMRWzgHEGg/?t=583) It seems that these models work super well in easily verifiable domains and coding and math have obviously been among them. How do you think about how, for some of these less easily verifiable domains, these models may end up scaling and being useful? I mean, they're getting better at that stuff too, but yeah, it's definitely harder. I guess in those domains, we're going to need either better ways of verifying or more human feedback loops.

[10:16](https://realalpha.app/v/atMRWzgHEGg/?t=616) Yeah, I think what's good is that we're seeing is like, even with the Gemini model series, they're able to follow much more abstract instructions. So being able to try and provide a reward signal over a qualitative piece of work, which maybe if a human were going to try and give a feedback signal, there would be a quite a broad set of like rubrics or grading criteria. or maybe there is even more like a even kind of simplistic sense of like what is good style what is like interesting and and so i i think part of the problem is really training models to like take on these like very broad criteria for what uh how to how to like take in a very broad set of criteria and then apply a reward signal once we have the reward signal we can train with reinforcement learning against it and uh and i think yeah we're already seeing that kind of like

[11:11](https://realalpha.app/v/atMRWzgHEGg/?t=671) makes makes sense it's not like an abstract thing anymore it seemed like a very abstract thing maybe a year ago or two years ago i'm curious a year or two ago like did you expect that to work or do you think this like whole path of of research was really more toward these like more easily verifiable domains i feel like you expect it to work one day and then it feels like there's a very there's a very complicated stack of things that we feel like we need to solve to get there and then usually the case it's like oh it turns out there was a much simpler path that's how i feel about it anyway yeah usually there are good surprises you know more good surprises than bad surprises you've released these models out in the wild um what are some