Anthropic makes reasoning harder to distill without breaking API use

Transcript

From the transcript Latent Space publishes with the episode.

1. 9:44

   I don't know how much this paper specifically caused a freak-out, because this is something that we knew [laughs] was a gap. Um- Yeah. So there, there's two separate things. Like- There's the Chinese model distillation- Yeah... and then this g- was a specific technique that was published that was high-profile. Yeah, exactly. I... We, we've known for a long time that chain of thought extraction is one of the most powerful ways that you can distill models.

2. 10:07

   And actually something like how do we preserve thinking blocks across turns and make that something that's more required is something that we've explored back for a while. And what we... where we were at was we always have this really, really hard trade-off that we have to make between user experience, like user experience changes that are maybe painful for users of the API versus protection against distillation.

3. 10:33

   And so how do we kind of walk that line and say we're preventing distillation? And maybe just to take a step back, the reason we don't want distillation is because the entire point of why Anthropic exists is 'cause we want AI to create positive outcomes in the world safely, and we run a lot of safeguards classifiers on the tokens that we're producing in order to make sure that they're safe. If someone distills our models and they run those models without safeguards classifiers, there could be bad outcomes in the world, and that's something that we don't wanna have happen.

4. 11:02

   So we don't want people to distill the models specifically for [laughs] the sake of, um, just, like, using them for malicious reasons or for bad reasons. And so it's always an interesting trade-off of, like, for all the people who are using the models for good, right, like we don't wanna make them have a harder time getting the intelligence out of the models, but we have to make some trade-offs in order to prevent that distillation.

5. 11:24

   And so we had been working for a while on, how do we prevent that chain of thought extraction? Like, what do we do about preserving thinking blocks in a way that wouldn't be too painful for people? And we kind of think about user experience hits in, like, two layers. Like, one is just, like, you have to make an integration change, and it's annoying, but you could still accomplish all the same things that you otherwise would've been able to accomplish. It's just a little bit annoying.

6. 11:46

   And then two is, like, actually you can't accomplish the same things that you could have accomplished previously, and that's the one that we're, like, really, really, really trying to not have happen. And so where we landed with the last couple models is a few incremental changes to have, like, enforce these preserve thinking blocks across turns, and now, like, starting to be more about, like, across, you know, like, if you're using multiple accounts and things like this, right?

7. 12:09

   Like, how do we make sure that We're, um, ensuring that you aren't able to take advantage of this chain of thought e- extraction with the models. And so we shipped a couple additional features to help with that. We worked actually with a lot of open source harness maintainers. We reached out to them, we talked to them, we were like, "You're gonna have to make these upgrades within your harnesses. We wanna make sure you have a good time," as well as obviously our big customers, like who are building harnesses on top of the API.

8. 12:35

   And for some of them, a really good example is they have compaction strategies that would've been broken by the need to preserve these thinking blocks. And so we shipped a new, um, compaction feature within the API to replace what they had otherwise been doing manually, but still works with preserved thinking. So it's just an example of like we're walking this line, but we didn't just like kinda YOLO out like- Mm-hmm...

9. 12:57

   now you have to preserve these thinking blocks in this exact way because we would've broken a lot of people, but we were able to take the time and work with customers and open source harness maintainers and give them what they needed to be able to make that upgrade.