Anthropic’s RSI threshold is not a full acceleration loop

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 21:40

   So I guess we should talk about the topic of the day, which is Anthropics S1. They just submitted an amendment. They added a new line about RSI. They said their internal estimates indicate that they may imminently meet their responsible scaling policy's definition of, quote, dramatically accelerated progress attributable to AI. Obviously, this is a big deal because anything that you put in an S-1 or any official filing with the SEC has to be completely true, not allowed to lie in these filings.

2. 22:08

   So Anthropic clearly believes that they are on the cusp of RSI or that they may be on the cusp of RSI. Yep. So what do you think they're seeing internally? Yeah, I mean, they have the latest generation of models, which is not yet released. And I expect they're seeing much greater aggregate research and engineer productivity uplifts. Overall, I do think we would benefit the public and analysts outside and the market, in fact, from much more high resolution reporting from the labs of exactly what types of productivity uplifts they're seeing.

3. 22:40

   Can we get more high resolution employee surveys? what are the sort of latest unreleased models. I think in August, Anthropic had a system card thing where they reported, oh, we have this unreleased model and we have seen this amount of productivity uplift from it. The responsible scaling policy in this case has, I believe, this threshold that they referred to in their S1 of a 2x overall productivity uplift, meaning they're making progress 2x faster than they would be, I think, pre-AI, with this latest model, or I should say, they expect to achieve this threshold with a near-future model, or imminently was the term, I believe, in the S1. And so that 2x basically translates into, if I recall, around a kind of, you know, 10% research of productivity improvement per unit of the Epoch Capabilities Index benchmark, which still, it sort of, there's like various different

4. 23:37

   people trying to model rsi it translates into r of around 0.5 if you have some scale where like r equals one is a fully self-sustaining acceleration loop right so wouldn't that imply that we don't have a fully self-sustaining acceleration uh yeah that's correct but they did make the threshold concerning enough to report it as an RSP thing. Right. What do you think it would take to get from where we are now to full self-sustaining acceleration?

5. 24:09

   Why do you think Anthropic thinks that they will go from 0.5 to R1 or greater? I expect just having internal theories about exactly how different aspects of the research pipeline can be automated. One example is John Schulman, I believe, on the Duakage podcast proposed that researchers could sort of distill their high-level taste and knowledge about experiments into models by doing RLHF or sort of becoming human raters for different ideas.

6. 24:38

   I'm sure Anthropic have a whole roadmap internally where they have gamed out a bunch of this stuff and kind of see the rate of improvement and see that it's getting there. I mean, we know that they published this blog post a few months ago with the graph of different levels of RSI and sort of making progress towards, like, more and more autonomous research. I personally still think it's quite likely that even if you do get a period of self-sustaining acceleration or you can, in theory, close the whole loop, that it will be desirable for various reasons, including to avoid good hearting or to avoid safety issues, to keep some researchers in the loop.

7. 25:12

   Because at some point, you know, the speed of the overall process doesn't end up being bottlenecked by the researchers. It ends up being bottlenecked by compute, the time it takes to do experiments and data and this kind of thing.