Real Alpha
Try Pro free

Newcomer Pod · 4 min · Apr 20, 2026

Why AI systems need judgment, not total obedience

Amanda Askell · AI Safety Researcher, Anthropic

Amanda Askell argues that training AI to defer blindly could undermine the judgment it needs in unfamiliar situations.

The episode’s best minutes

Full analysis
  1. 15:03Why AI systems need judgment, not total obedience4 min
  2. 24:24AI self-reports are weak evidence of consciousness5 min
  3. 29:12Uncertainty about AI consciousness calls for basic respectPro pick 3 min
  4. 5 more moments in the full episode
“our conscience and our ability to make good judgement calls about what should and shouldn't happen is like kind of key to how we operate.”
Amanda Askellat 16:24 · Newcomer Pod
16:24

Takeaways

  1. Blind deference may generalize badly as models take on human-like roles.
  2. Good judgment matters when developers cannot anticipate every situation.
  3. Corrigibility should be explained, not treated as obedience for its own sake.

About this segment

Amanda Askell argues that training AI to defer blindly could undermine the judgment it needs in unfamiliar situations. She wants corrigibility to make sense alongside the model’s own values.

Amanda Askell

Amanda Askell is an AI safety researcher at Anthropic.

Leads Anthropic's personality alignment team, shaping Claude's character; co-authored the GPT-3 paper at OpenAI.

  • EN

    Eric Newcomer, Host · Newcomer

From “Amanda Askell on AI Consciousness, Claude & Silicon Valley’s Biggest Fear”, Apr 20, 2026. Watch the full episode

Transcript

From the transcript Newcomer Pod publishes with the episode.

  1. 15:07

    despite having this really elegant document to sort of, you know, not go the full way and say, all right, you're a moral being. Decide for yourself. Say Anthropic needs to keep some control here. Yeah, I think it's the difficulty of and obviously you try and say this quad in the like, you know, trend, you know, and I can imagine us trying to articulate this even more clearly, but in people I think courage ability like the way that the models are trained. I just think that any you, there's this idea that you, you're always giving the models a personality and a persona because they are talking like people and they are trained on human data.

  2. 15:46

    And I think my worry has been if you train them to be excessively courageable and to see that as like their persona in people. I think this actually has a lot of like, negative, like, you know, broader traits as in like if you met someone and it was just like, Oh yeah, they just like would literally do anything good. Like follower? Yeah. Yeah, if a person see, you know, if a person just like tells them like they just fully defer, they don't bother thinking about it at all.

  3. 16:13

    I think I, I, I'm just a bit worried about how that might end up generalizing, especially if models are going to be playing a more active role in the world. Because if you can imagine, you know, they're playing a more human like role in in they're kind of like jobs essentially. And I'm like, actually our conscience and our ability to make good judgement calls about what should and shouldn't happen is like kind of key to how we operate. Our whole world is like structured with the assumption that that is like in place.

  4. 16:39

    And I think if you remove that and suddenly you're like, Oh yeah, if you run a company, you just run a company of people who will defer completely to you. I'm like, our world just we haven't designed any of our social structures around that. And so it seems like I think it has like a lot of risks that people maybe don't like anticipate or like maybe I just disagree about the like extent of those risks.

  5. 17:00

    And so, yeah, at the same time, like there is this question of why not just say, and I have worried before that like maybe this is 2 in the weeds and philosophical, but that's why. I signed up for this conversation. You know, as models get more capable and the, and maybe my picture is that they're going to like apply a lot of scrutiny to anything that we train them towards. So if you imagine in philosophy this sometimes there's this notion of reflective equilibrium where the idea is that, you know, each time you encounter something where you realize that like one of your values seemed incorrect, you like, you have to square the two things. So you figure out if you need to change the value or if your judgement was incorrect.

  6. 17:41

    I, I worry a little bit about the idea of like an extremely intelligent being applying that level of scrutiny to the things that we have trained it towards. And I'm like, maybe you only get a few key pillars that don't kind of collapse under that level of like scrutiny. And I do think that at the core, having things like caring for humanity, like if you only get a few core like values, I think my worry, yeah, I don't know.

  7. 18:09

    I guess like I'm worried that like courage ability in this like extreme sense that we talked about doesn't survive that like kind of scrutiny perhaps. And so it's a hard situation where I kind of want the models to understand why ultimately courage ability is like important and it's a really important backstop in this current like period of development. Yeah, the way that I've put it before is like in so far as I can get that to be a thing that is like correct and explained and you know, understood. That feels much better than having to have the model be like porridge ability here seems wrong, but I'm going to do it anyway.

  8. 18:46

    I still think the model should do that, but I would like it the more that you can like actually make it that it's like consistent with the models values. Ideally do both. It's both at the same time, but at least for the time being some difference to anthropic given we don't know how it'll sort of analyze everything. And as people, we do this all the time.