Current harnesses do not make models learn strategy on their own

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 54:18

   is that it's maybe a harness-solvable challenge versus a model-solvable challenge, and maybe, you know, no one's focused on that. Like, with the right set of memory structures and, like, some kind of sidecar heuristic-y thing, like, you know, we can probably make a big step improvement in AI's ability to, like, you know, improve game-to-game, but it's not, it might not require dramatic model changes.

2. 54:51

   Do you think about that? And is, do you have, you know, in this domain or in other domains, like formal or structured ways of thinking about, you know, what, you know, model versus harness, you know, granted this whole model versus harness thing in and of itself. is, you know, practically brand new, but it's proving to be important in, you know, distinguishing, you know, where these improvements come from.

3. 55:22

   Yeah, definitely. We've done some experiments on this for the board game case in particular, where we've tried a very simple harness, first the first party harnesses like Claude Code or Codex. We've tried multi-agent setups. we've tried like we we give them a you know note-taking tools and say you know make it clear like okay your your goal is to learn the game so you can do really well like in your final plays of the game so explore take notes figure like we try to prompt them well i i mean we want to find good results if we can because it's like that's a big deal for the world uh the game is a test bed but real jobs would also you know benefit significantly risks would also go up if this capability you know you test in the lab is it a great hacker you find maybe it is maybe it isn't but you find that it isn't and yet it can learn on the fly then okay once you deploy it like look

4. 56:18

   out um so anyway we we've tried pretty hard and have mostly not found anything that cracks this uh that cracks this on the fly learning there's been some general research about this as well not just us about uh you know people taught like there was uh people still use these so-called skills where you like just write some notes to an AI system of, look, when you do work I ask you to do, you're going to need to use this tool or produce this kind of output.

5. 56:46

   And like, here's my notes on how you could do that. Well, so you don't have to like learn it on the, like pick it up fresh every time. So use these skills. And these show some improvement, but then there's like a plateau like pretty quickly, at least in the research I'm familiar with of like, you can get some improvements. My senses, this is very loosely speaking, it's like low-hanging fruit. Like if you're telling it, oh, you're going to have to navigate this like finicky website that my company makes me use.

6. 57:14

   And like just, you're going to get stuck and like you have to look in this menu and that's where the button you're looking for. Like straightforward advice that's like obviously useful. But when it comes to higher level strategic thinking, kind of hard to bake that into the harness or like where the board game plays, like has this, you know, sort of brings this out where it's like there's not any perfect advice that like you have to find the advice for yourself in a sense like you have to learn the game like so i will say if we give it extremely detailed strategy guides for here's how you beat this game like here's everything you could possibly want like it's a cheat by human state like no one would play the game that way they do very they do very well they do much better than if they basically have to find that

7. 58:02

   The question is, do they, right, creating the strategy guide. They can't create the strategy guide for themselves, which is, like, pretty interesting. So, like, we do expect eventually they'll get better at this. But it's, again, it's a big deal if that happens, because now it means, okay, go back to that first benchmark suite. Okay, they're not great at making charts for us or coming up with data insights for us, you know, without guidance.

8. 58:28

   uh but now put them in on the team for a little while and let them learn on the fly and maybe they do become good so you sort of see these as a combination like if on our board game where it's easy to measure they're not getting better over time then it's okay for us over in the real world like human in the loop evaluation uh studies for us to just do one try and say okay well they weren't great and we don't think they learn on the so that you know but but really you know those go together and is the board game a synthetic one that you created for this challenge or is it a