Interpretability can monitor, diagnose, and shape model behavior

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 53:26

   that really know how to do this. Great. Let's go a bit deeper in some of what you alluded to towards the beginning of this conversation, which is a fundamental question of like, okay, We're able to now go inside the mind, quote, end of quote, of the model and sort of have a glimpse into how they function and what they do. The next question is, so what do we actually do now? Like what actions can we take to minimize all the problems and rewater hacking and all the negative stuff?

2. 53:59

   What's the range of things we or labs can do? I think it comes in three, maybe like tiers. The first is monitoring. So how can you just detect bad behavior, you know, before it even happens such that you can just stop the generation or, you know, another common intervention there is you can insert a text prompt in. So like, let's say the model is about to do some dangerous cyber action. You can then just insert into the prompt of the model, like, hey, be really careful here.

3. 54:39

   Don't take a dangerous cyber action. In real time? In real time, correct. So this is like some type of prompt steering. And then you can then reduce the model's chances of taking a harmful cyber action because you've intervened directly into the model. So that's step one. It's like, how can you monitor for this behavior? And you already get quite a long way. Step two is kind of at-scale offline anomaly detection and debugging.

4. 55:10

   So the problem statement here is given a vast amount of logs and traces, how do you actually surface the anomalies from all of these traces, all of the problematic behaviors, and then reverse engineer the model to figure out where mechanistically the model has actually gone wrong. So it's like figuring out what the problems are in the model so that you can fix them. And then the third step is what we call intentional design. We think this is really the holy grail of of ai alignment of maybe all of ai training as well where you can actually control generalization in the model and you only get the good stuff in training and none of the bad stuff. So how can you steer and guide the training process so that you can actually design these models with intention and rather by trial and error? So maybe just one little mini

5. 56:12

   example here where let's say you understand the concept of reward hacking. And so you're training the model and you realize that during this gradient step, the model was about to reinforce its idea of reward hacking. You don't want that to happen. You actually want reward hacking to go down. And so how do you turn that like increased reward hacking signal and either like remove that entirely and reject that update or like reduce that behavior in the first place? That I think is really what we mean by how do you steer and guide training? How do you steer and guide back crop how can you actually imbue the values the morals of the model that you want with the according to some type of constitution that you that you specify so that instead of having an absent teacher you actually have a a very moral and values aligned teacher teaching your model

6. 57:15

   to to be good so you literally give gradient descent a choice that's right yeah we want to give gradient descent a choice so what's the current state uh of this is intentional design uh something that you're working on and that's like the next like uh whatever one two three years of of research or is there something that's working today like where what's the state of the art we have a couple rudimentary techniques that work in this like umbrella of intentional design so there's two things that we've published so far um but i'll also hint that there's a lot more exciting stuff coming just around the corner. We have some very, very good internal results here to help with intentional design. But the two techniques that we've published are, one, reinforcement learning from feature rewards. You can essentially take a probe, and you can help,

7. 58:08

   you can optimize against that probe to remove, we showed that we can help remove, like, hallucinations in Gemma, using this as a reward signal. The setup here really matters, though. You can't just naively train against a probe, a probe monitor or a probe concept. Otherwise, that just moves this concept into some other part of the model. So you need a relatively sophisticated technique in order to do this correctly. So that's another paper of ours, reinforcement learning with feature rewards. I thought that was like a really interesting first step, but it was kind of like a more simple and rudimentary setup. Another idea is predictive data debugging, where you intervene from the data side. So the problem statement there is how can you predict what your model will learn from a data set before the model even trains on it? And then

8. 59:07

   And what ends up working best is some type of clustering technique where you can cluster data points according to what they will teach the model. And you can then just remove the data that you don't want. So we're able to find pockets of data in these public data sets that were quite surprising. Like one of these pockets of data was physics sycophancy.