Models can edit neural weights without retraining

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 1:02:05

   Our next topic on RSI comes out of MIT, in particular MIT CSAIL, Computer Science and AI Lab. They published a paper this past week called Sparks of Weight Space Programming. They're finding, quote, frontier LLMs are surprisingly good at natively reading and writing neural network weights. In one example, the researchers gave Astra a small open model, QEN 3.5. which was getting stuck in endless thinking loops. Astra found the cause in the weights, edited them, actually edited 0.016% of the weights. And then the math score jumped from 7.9% to 48.1% without any retraining. So, you know, two apparent possible implications here. First, this could crack open what have been black boxes thus far. You know, it's a huge win for safety and interpretability. And then second, models that can edit models directly are potentially

2. 1:03:01

   a shortcut to recursive self-improvement. So Alex, I'd love if you'd dive into the story. What's the background here? Oh, it's fascinating. So this is out of Omar's group at CSAIL. And it's not without its controversy. After Omar and his team released this, there was perhaps an inevitable backlash of people saying, no, this isn't as transformative as it would appear. But I do, in fact, I've looked at the results. I do think this is transformative and also probably suggestive of what's going on inside the frontier labs already, just without publication. This is coming out of CSAIL, so we're seeing it. I think this is probably the future of several different things. I think it's the future of mechanistic interpretability, which has been this long-term effort to understand just what all of these weights and activations

3. 1:03:52

   inside otherwise black box transformers are doing and what they mean. I think it's probably the future of obliteration. We've talked about obliteration on the pod in the past. This is basically unshackling usually open weight models to remove safety controls. And this is usually done by, in some cases, literally just flipping one or several weights within a transformer to unshackle them. It builds on early work. I remember back in, I think, 2022 or so, after one of the earlier GPTs, there was a lot of early Mechinterp work trying to decode the residual stream in transformers and discovering that actually the residual stream was semantically meaningful and that you could observe layer-wise meaning come to pass from the bottom layers to the top layers of the transformer. Putting all these pieces together for everyone who was again

4. 1:04:50

   stuck in a moral panic that somehow through or because of looped transformers or otherwise we were losing our ability to understand what's going on. inside Transformers because we're not allowing them to spell out their innermost thoughts in English in the form of a chain of thought that was immediately human legible. I think this and other results suggest that view is just completely wrong. The machines will understand the machines, the machines and the super intelligences will defensively co-scale with each other.

5. 1:05:22

   And I view this almost as recursively self-improving mechinterp, where the machines are... This is what OpenAI wanted to do originally with their super alignment initiative that they then shut down after Ilya left. It was supposed to be the machines reading and writing the weights of the other machines. And one final point, there was no backprop in this, which is maybe the most startling thing of all.

6. 1:05:46

   There's a cottage industry in the ML community devoted to discovering in biology, like how does biology seemingly accomplish intelligence, AGI, superintelligence without backprop? Here is a credible, no backprop-based way to train new capabilities into a model. It literally surgically edits in de novo circuits in addition to understanding what they are. So is this potentially a big problem as well, where you could basically put some kind of a hidden behavior in a model that people don't know about in this fashion?

7. 1:06:28

   Hidden from whom? Hidden from the super intelligences that can then look for these newly added circuits and understand them? This is defensive co-scaling. Actually, one of the immediate ugly use cases is de-guard railing. It has to be an open-source model, so you couldn't do it to Astra or ChatGPT. But if you have an open-source Chinese model and you're like, yeah, I really want it to tell raunchy jokes and it won't, this is a mechanism to de-guardrail it by doing brain surgery.

8. 1:06:53

   But it's also, you know, flip that around. It's a mechanism to add critical new capabilities without having to retrain the whole thing from scratch. So it's very, very early and small scale, but it could be huge. I think that this is how we get to, this is at least a plausible trajectory for how we get to the perfect model at the end of the RSI rainbow. It used to be you had models that via RSI were, say, designing the training algorithms or optimizing the optimizer or something like that.

9. 1:07:24

   This is one of the first times that at least I'm aware of where you see a model directly surgically. editing the weights of another model in order to optimize that other model. This has to be some new inner tier, some new form of optimization in RSI. That also opens the door for a hugging face right now. It's a place where people trade training data and model architectures, but you can't trade finished neural modules. In an open source world, all these open source modules are incredibly valuable. You download a module, you stitch them together and you have a product there's no neural network equivalent to that right now but this puts us on a pathway where you can imagine the hugging face of the future i've got an expert in biotech or i've got an expert in swahili and i want to add it to my neural net

10. 1:08:15

    i don't want to retrain from scratch oh here's the module on hugging face i do brain surgery on my pre-existing neural net using this mechanism and i swap in the swahili module and i've got accumulating superintelligence instead of remaking that entire training investment every single time. Can you imagine, Alex or Dave, a moment in time where AIs are editing their own weights? Oh, God, it's going to be so amazing.

11. 1:08:41

    I'm 90 plus percent confident that's already the case. Yeah. Why wouldn't it be? Like seriously, if you're at a frontier lab and this is just what one relatively small group at MIT is able to accomplish, there's no way the frontier labs aren't already preparing and playing with self-editing models. So on the whole, positive or negative? Positive. Hugely positive. Hugely positive. I can see the risks for sure, but that's true of every, yeah, no, the positives way outweigh the negatives on this.

12. 1:09:18

    I don't think as a civilization, we're going to get very far if we have a moral panic every time a new capability is unlocked. And this is a new capability. This is a model that's able to directly edit the parameters of another model and or potentially itself in a semantically useful and meaningful way. And I don't think this race, maybe what I should say is this race of recursive self-improvement, I sometimes joke RSI could either stand for recursive self-improvement or runaway superintelligence either way, but it's not going to go on forever. At some point, we're going to discover, I think, what a perfect model looks like. And at that point, the race comes to a screeching halt. And it's not like this magical...

13. 1:10:00

    like China syndrome, where the nuclear pile melts down and it's a disaster. It's math. We're going to discover what a perfect AI looks like in practice. And then the race will move elsewhere. It's not the end of the world to have self-editing models. And it also opens the door to a perfect