Unsupervised Learning · 2 min · Mar 17, 2025

Qualitative tasks need broad reward criteria

Jack Rae · Research Director, DeepMind

Jack Rae says models can be trained against broad rubrics for qualities like style and interest.

In this episode

19:34 Inference is cheap enough to justify much more thinking6 min
25:29 AI math progress depends on asking better questions6 min
31:48 Research teams need both bottom-up and top-down bets8 min
9:39Qualitative tasks need broad reward criteria2 min
15:57 Agent research depends on choosing the right environment4 min
42:03 Use one model unless specialization earns its cost2 min
1:06:30 Inference-heavy AI can use more distributed compute3 min
58:15 Hallucinations made entertainment a natural early use2 min

“take in a very broad set of criteria and then apply a reward signal once we have the reward signal we can train with reinforcement learning against it”

Jack Raeat 10:54 · Unsupervised Learning

Takeaways

Rubrics can describe qualities such as style and interest.
Broad criteria can produce a reward signal for training.
This extends reinforcement learning beyond easily verified tasks.

About this segment

Jack Rae says models can be trained against broad rubrics for qualities like style and interest. That offers a path for reinforcement learning beyond tasks with automatic right answers.

Research Director, DeepMind

Led Gemini Thinking, co-led Gemini pre-training, and contributed to Gopher and Chinchilla language-model scaling research.

Jacob Effron, Host · Partner, Redpoint
Growth investor in Abridge, Ramp and Physical Intelligence; ex-Flatiron Health.

From “Noam Shazeer and Jack Rae: Scaling Test-time Compute, Reactions to Ilya & AGI”, Mar 17, 2025. Watch the full episode

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

9:43
It seems that these models work super well in easily verifiable domains and coding and math have obviously been among them. How do you think about how, for some of these less easily verifiable domains, these models may end up scaling and being useful? I mean, they're getting better at that stuff too, but yeah, it's definitely harder. I guess in those domains, we're going to need either better ways of verifying or more human feedback loops.
10:16
Yeah, I think what's good is that we're seeing is like, even with the Gemini model series, they're able to follow much more abstract instructions. So being able to try and provide a reward signal over a qualitative piece of work, which maybe if a human were going to try and give a feedback signal, there would be a quite a broad set of like rubrics or grading criteria. or maybe there is even more like a even kind of simplistic sense of like what is good style what is like interesting and and so i i think part of the problem is really training models to like take on these like very broad criteria for what uh how to how to like take in a very broad set of criteria and then apply a reward signal once we have the reward signal we can train with reinforcement learning against it and uh and i think yeah we're already seeing that kind of like
11:11
makes makes sense it's not like an abstract thing anymore it seemed like a very abstract thing maybe a year ago or two years ago i'm curious a year or two ago like did you expect that to work or do you think this like whole path of of research was really more toward these like more easily verifiable domains i feel like you expect it to work one day and then it feels like there's a very there's a very complicated stack of things that we feel like we need to solve to get there and then usually the case it's like oh it turns out there was a much simpler path that's how i feel about it anyway yeah usually there are good surprises you know more good surprises than bad surprises you've released these models out in the wild um what are some