Running a company exposes AI’s gaps in reliability

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 12:41

   think that like when you talk about how you know jaggedness could persist over a relatively long period of time are you imagining something like a model that is like wildly extremely superhuman at math and theoretical physics research and such that still can't really write a novel or something yeah i think novels are probably yeah let's see so models are already pretty close to superhuman on math um in some ways they are superhuman in other ways they're still falling short uh writing i expect them to be superhuman novel writing uh reasonably soon as well uh just because it's the kind of task that doesn't take that much feedback from the world the sort of thing that feels more difficult is running a company for long periods of time uh and when you're running a company especially a larger company

2. 13:38

   you just have this enormous attack surface where you know one mistake uh can be exploited by a bunch of adversaries like hackers your uh your rivals you can mess up and then be in trouble with the legal system in a way that shuts down the entire company. So all of those things prioritize a certain level of robustness that the models don't currently have. And my sense is that it could be pretty difficult to get the models actually reliable on a task like run a company for six months and make sure it's profitable and doesn't end up in legal trouble. Yeah, it does seem to me on the topic of people jumping to conclusions beyond where the evidence is warranted, One such area is the idea that, you know, every AI capability is just emergent.

3. 14:24

   And, you know, if you automate AI research, if you have models working to overcome their own shortfalls, then the areas in which the models are jagged in a bad way will like just go away, basically. And this seems to me to be like plausible, but far from certain. Yeah. I also want to distinguish between capabilities that exist within the models and capabilities that we're actually able to access. So right now, I think there are probably a bunch of capabilities that the models have that we just don't know how to get out of them.

4. 14:58

   You could call this an alignment problem or an elicitation problem. A historical example is just simple things like saying think step by step. were able to elicit capabilities from early language models that, you know, were overlooked for quite a long time, I think one to two years. And so, yeah, I feel like there's going to be some point at which capabilities and alignment as a concept become harder to distinguish because alignment is, in fact, the bottleneck on eliciting the capabilities.

5. 15:32

   maybe for example models will only demonstrate certain capabilities when they trust the person who's trying to elicit it from them um and uh that might in fact already be true it's a little hard to say the models are pretty aware of uh you know the context in which they're being prompted who's asking them to do stuff and i wouldn't be surprised if um if models for example if claude for example, had a judgment of Anthropic overall that helped shape which capabilities it was displaying.

6. 16:07

   And if Anthropic behaved in a more disreputable way or a more trustworthy way, then that might change the capabilities that Claude is willing to showcase.