Single rollouts can train agents without replay environments

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 13:07

   Right now, it's mainly you bring traces and reward and some notion of environment. The two parts here that require a bunch of manual work is the environment, actually constructing this thing that matches the real production. And a lot of the time, it's actually impossible to create a perfect environment, right? And then the second part is reward. A lot of the times, a legal workflow or investment banking workflow doesn't have some perfect verifiable reward that you can train off of, right?

2. 13:39

   And so these are the two areas that we're really pushing on the research side. So for the environment, there's an algorithm called GRPO. That's the main kind of post-training algorithm that exists right now. What you need to do there is you basically, like imagine you have a player that's in a video game, and it's basically going around and hopping around a bunch of platforms. and then at the end of the day, you get some reward.

3. 14:01

   What's needed here in GRPO is basically comparing the same task, the same level of the video game, and you need to play that out eight times or 16 times, and we call that a group. Now, the thing is, with humans, a lot of the time, we don't need to replay the exact same environment over and over again. We're just in the world experiencing and doing one rollout. And so the goal is, could we modify GRPO as an algorithm to get to a world where it is what we call group size one instead of needing a group of 16.

4. 14:30

   And so we've made a lot of strides here where now if you can take 10,000 trajectories and basically they're all from different tasks, we can actually learn from just that one behavior or that one rollout. And so that's one algorithm that we actually released a blog post developed is going into this group size one paradigm. And so that actually eliminates the need for an environment and all of this infrastructure that needs to be built up.

5. 14:53

   the second part is for reward and so this is another factor where let's Say I have, again, 10,000 traces, the default is that I need to go and label every single one of those traces to say, here's verifiable reward, or here's what good looks like. And this is tractable, it's just a lot of work. And so now our goal is, how can we actually reduce that amount of time needed to take this notion of what good looks like and do that in much less amount of work?

6. 15:23

   So what we've realized is, let's say I have 10,000 traces. if I can smartly select 100 of them that are basically representative of all of these broader 10,000 and then have the customer or the user basically densely say, okay, here's what good looks like. Here's a rubric or verifiable criteria for just those 100 traces. And then what we can do is basically train a metagrader that learns from those behaviors of the 100 to the broader 10,000.

7. 15:51

   And with that, you get a really clean interface that allows you to just propagate this reward across the board. So I think obviously it's not the perfect solution in that direction. The ideal goal actually is that you have some sort of observability interface where you're like, here's the five things my agent's good at. Here's the five things it's bad at. And I just circle those five things and hit train.