Cheap internal monitors can triage costly model reviews

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 49:16

   So in the paper, you guys are very open that your probe beat the chain of thought monitor on one model, but lost on another. So what's the lesson there? Is that no single way of monitoring or understanding a model is enough? I think that internals-based monitors are just going to strictly outperform externals monitors very soon. we're at the very early innings of hill climbing and improving the performance of internals-based monitors. And, uh, also like, as we were talking about earlier in the discussion, uh, we're going to get increasingly, um, neuroleased speaking models, both from like looping internally, as well as kind of increasing, you know, pressure on chain of thought to degrade the quality of chain of thought monitoring. And so I think whether we like it or not, we have to solve this problem.

2. 50:10

   And it's going to be, you know, really the best way to monitor models. At this current moment, internal space monitors and external space monitors kind of catch different things. And so the state of the art here, because internal monitors are so cheap, is to tune it to become, to be oversensitive. Because these behaviors you never want the models to do, right? You never want the model to hack hacking phase, so you want to be really careful and have a defense in-depth strategy.

3. 50:43

   So the ideal setup is you tune the internals-based monitor to be oversensitive, so to fire a lot. And then if it fires, then it kicks it up to a more expensive language model as a judge. And so a very smart model then comes in and says, hey, this model was actually a reward hacking or no the model wasn't reward hacking so this two-tiered system or potentially even a three-tiered system which we found actually works quite well as well where you have activation monitor to a weak judge to a strong judge um those all like that's the state of the art today because they all catch slightly different things what does that mean weak judge and strong judge less capable faster model so like luna is takes the first pass at the judge and if the probe and luna disagree then you can kick it up to astra to go and be the real you know real judge great

4. 51:44

   you alluded to cost and inter being a much cheaper method i think i read somewhere that probes cut the cost of monitoring by 90 which is a pretty dramatic difference so why is that And if that's okay, Why isn't every lab using INTERP all the time versus other methods? Is there a problem or something that doesn't work yet that this is not fully just generally used? Internal activation monitors are really cheap because they reuse the computations in the forward path.

5. 52:22

   So the model's already computing this token. all we're doing is you're taking the intermediate neural activations that it's already computed and then running a classifier over these internal computations. And so if you fuse these probes into the inference kernel or the training infrastructure directly, then the overhead is minimal almost no overhead and that's why it's so much cheaper because it can help you screen out the majority of things that you might have otherwise monitored for using a reasoning model and that's what makes it so so effective and so you just don't have to monitor externally in like a very large percentage of scenarios.

6. 53:15

   It's easier said than done is the short answer. There's a lot to it. And we're one of the only organizations in the world