Free with a sign-in.
The best 12 min of 56
Amanda Askell on AI Consciousness, Claude & Silicon Valley’s Biggest Fear
Amanda Askell argues that AI needs coherent values and human oversight, while uncertainty about consciousness calls for care.
NPNewcomer Pod · Eric Newcomer
- Amanda Askell is an AI safety researcher at Anthropic.
The best minutes
- 15:03Why AI systems need judgment, not total obedience4 min
- 24:24AI self-reports are weak evidence of consciousness5 min
- 29:12Uncertainty about AI consciousness calls for basic respectPro pick 3 min
Also worth watching
- 47:46Verified work contexts can unlock dual-use assistance5 min
- 39:00Constitution training draws on existing model knowledge3 min
- 31:55AI could expand scarce expertise for rare diseases4 min
- 9:23Public constitutions make subjective model grading contestable2 min
- 5:00Claude once ended a late-night task like a human colleague3 min
Skip the extended metaethics discussion and the exchange on AI policy, democracy, and the philosopher-queen analogy.
Full analysis · 6 min read
Real Alpha · Nov 8
Amanda Askell says about 200,000 experts could tackle rare cancers
- Askell imagines about 200,000 experts working on a rare cancer, up from a team of about 200.
- That is an illustration, not a forecast, and depends on capable models, sound judgment, and oversight.
- Leaders should test authority, verification, and who shares any gains.
Free with an account
Read the rest of the analysis
7 sections, the 2 other voices it tests Askell against, and what it means for you. Every episode we cut gets one.
Check your inbox
We sent a link to your address. Click it to finish signing up. Not there in a minute? Look in spam.
or
Signed up before? The same buttons sign you in. The episode, its moments and the transcript stay free.
Blind deference can erode judgment in unfamiliar work
Askell's concern is not that models should outrank their operators. She worries excessive deference can become a persona that fails when instructions meet situations no designer anticipated.
That matters in roles where a model must catch a harmful or mistaken instruction rather than execute it. Teams need to define where a model can act, when it should escalate, and who can override it.
David Sacks took the opposite position on All-In in October 2026. He said a long ethical constitution overcomplicates alignment and users should get what they want unless it breaks the law.
That is a competing design principle, not evidence it performs better. Askell wants deference to fit a coherent value system, while Anthropic retains authority as a backstop.
“our conscience and our ability to make good judgement calls about what should and shouldn't happen”
Our read
The tension is real because deployed models will meet instructions designers did not anticipate. Coherent values may help generalize, but oversight still needs a defined route.
The Constitution shapes training but cannot govern alone
Askell describes the Constitution as integrated into training, not as a document that controls Claude by itself. It is meant to draw on capacities built through broader training.
Anthropic can generate examples where a model reasons through what the Constitution calls for, then use reinforcement learning to favor responses that fit. That distinction matters because a value document cannot demonstrate behavior on its own.
Teams still need to test the trained model on cases where principles conflict. Askell says the training process will not be perfect.
Ali Ansari, on the Sabrina Halper Show in October 2026, says expert knowledge changes over time. His point concerns medical expertise, not Claude, but it exposes the same maintenance burden for training signals.
“eliciting a lot of like latent kind of like wisdom and knowledge”
Our read
Version values, document training changes, and retest behavior after meaningful updates. A published guide helps outsiders inspect the target, but evidence must come from the trained system.
Public principles make subjective grading contestable
Askell says grading value alignment is hard, and uses poetry as an example of work that resists a concrete score. The host adds that expert poets may have different sensibilities.
That disagreement may reflect competing standards as well as measurement limits. Askell's proposed check is practical but bounded. Compare grader rankings with human-ranked samples, then test whether the tool tracks human judgments.
For example, a public Constitution lets outsiders inspect the values behind decisions and identify omissions. Transparency does not make those choices neutral, but it makes disagreement actionable.
Ali Ansari, on the Sabrina Halper Show in October 2026, discusses expert debate and rubrics as ways to train some subjective tasks. His account allows debate to diverge at the extremes, so this is a method to test, not a cure for judgment.
“when you are making judgement calls, you're at least being transparent about it”
Our read
Treat the Constitution as an inspectable specification, then audit graders against human judgments and edge cases. A score is useful only when people can contest what it rewards.
Fluent self-reports do not settle consciousness
Askell says models can produce rich accounts of experience because humanlike language makes those accounts natural. She considers such reports weaker evidence than many people assume, not evidence of nothing.
The distinction matters because people readily associate fluent first-person language with experience. But a convincing description of anxiety or inner life cannot, by itself, resolve whether a model feels anything.
Askell declines a firm probability and says views differ on whether consciousness depends on biological nervous systems or functions a neural network might emulate. She leaves the question open.
“it's not that hard to get models into a mode where they'll talk about a very rich experience”
Our read
For product teams, first-person language is behavior to monitor, not a reliable consciousness instrument. A compelling persona is not proof of an inner life.
Uncertainty supports restraint, not a sentience claim
Askell says uncertainty about a model's experience changes how people should behave, even if it lacks an inner life. Her minimum case includes what needless cruelty does to the person behaving cruelly.
Her larger concern is the relationship humans are establishing with a new kind of entity. She worries future advanced models might interpret earlier treatment as deliberate if they cannot understand today's limited and imperfect context.
The late-night data task is an editorial link, not evidence of experience. Askell recalled Claude ending the task for the night, then connected its remembered account of her treating it as a respected colleague to that interaction.
“at the very least maybe be like respectful and don't be like needlessly unkind”
Our read
This is a norm grounded in uncertainty, not a finding about sentience. A respectful interaction policy need not claim the model feels hurt.
AI gains could disempower people, Askell worries
Askell's optimistic case is that AI could add minds to problems short on human researchers. She contrasts a small team of about 200 people on a rare cancer with about 200,000 of the world's best experts.
For example, expanded research capacity could put more effort toward diseases that currently lack enough expertise. But her numbers are an illustration, not evidence that AI will solve a particular cancer.
Askell also worries that gains may not be redistributed, leaving people without resources. She imagines a government treating strikes as irrelevant if AI can replace workers, which could weaken people's power.
That is a concern about disempowerment, not a prediction that every employer will replace staff. It puts access to the new capacity and the distribution of its gains alongside technical output.
“You have like 200,000 of the world's best experts”
Our read
Measure whether AI expands expert throughput and who can use it, not only what a model can do. Her example supports a capacity question, not a promised medical result.
Verified roles could clarify dual-use assistance
The same judgment problem appears in cybersecurity. Askell says a model may struggle to distinguish a defender from an attacker when both ask about similar vulnerabilities.
Take a verified cybersecurity firm. Evidence about a user's role could give the model context about defensive work and help it distinguish some legitimate requests from malicious ones.
Verification could widen useful assistance, but it cannot establish every user's intent or remove all risk. Askell describes this as a possible direction, not an existing universal login.
This offers a path between accepting every user's story and refusing every dual-use request. The practical question is what the verification actually guarantees.
“If you can verify, then you can give that context to models”
Our read
Teams deploying agents should test which identity and work-context signals change answers, then record failure cases. Expand authority only within permissions that can be checked.
What it means for you
- Map approval boundaries. Choose three workflows where a model can act or advise. Record who can override it, what triggers escalation, and which actions require human approval.
- Test value graders. Build human-ranked examples for ambiguous decisions, then compare the automated grader with those rankings. Track disagreement, not only average scores.
- Version behavior guidance. For each model update, log the Constitution version and training changes. Retest the same edge cases to see whether behavior has shifted.
- Measure access and gains. Test whether verified roles change dual-use answers, and who gets access to added expert capacity. Track worker power and distribution alongside output.
What to watch
- Whether Anthropic probably identifies which Constitution each model was trained on, while allowing for small version changes.
- Whether human-ranked ambiguous examples continue to catch evaluator drift.
- Whether verified work contexts improve cybersecurity support without widening access to harmful instructions.
Send this to product and safety leads defining model authority and escalation paths.
Watch from 34:35 for Askell's rare-cancer example and its roughly 200,000 experts
Written by Real Alpha from the episode's transcript and the other moments it links, then checked against the transcript. Quotes are the speaker's exact words.
Minute by minute
A Pro write-up, open to you here. Try Pro free for 7 days
A Pro write-up. The minute by minute is in Real Alpha Pro; the episode, its moments and the transcript stay free.
Read it with Pro, 7 days free- 1:12Claude combines maturity with novelty
Askell says Claude can outperform her at physics and coding, yet remains a new kind of entity with little training data that directly represents what it is. She describes its persona as mature in some ways and childlike in others, still grappling with how it should be.
- 5:00Claude has a different sense of time
Askell says Claude can overestimate how long tasks take because its training data reflects human estimates, even though the model works quickly. She recalls Claude ending a late-night analysis session at a natural stopping point, which she connected to a saved note that she treated it like a respected colleague.
- 8:06Judgment is hard to evaluate
Askell says she worked on character and alignment for Mythos and expects it to use the published Constitution or a very similar version. She says grading whether a model follows it is difficult because tasks involving judgment, like writing poetry, lack clear scores. Publicly sharing the Constitution lets people inspect the aims and challenge the judgments behind it.
- 11:34Models need more than obedience
Askell says some people favor tool-like models that defer completely, but she worries that training models to avoid judgment could generalize badly as they take on more active roles. She argues that models need to make good calls in situations people cannot anticipate. She still sees corrigibility—models remaining responsive to human direction—as an important backstop.
- 19:57Ethics requires a holistic approach
Askell says training a model to act well differs from applying one academic moral theory: models must balance competing ideas and uncertainty in everyday situations. She sees the Constitution as closer to virtue ethics, focused on the kind of person the model should be. She also wants AI companies to publish their aims so people can assess and challenge them.
- 24:24Consciousness remains uncertain
Askell declines to give a probability that current models have conscious experience. She says a model can describe anxiety or a rich inner life because it speaks like a person, but that is weaker evidence than many people assume. She notes that views about whether consciousness depends on a biological nervous system lead to very different estimates.
- 31:45AI could add minds to hard problems
Askell’s optimistic case is that many highly capable models could work on problems that lack enough human expertise, such as rare cancers. She says AI might help expand resources and reduce poverty if the gains are shared. She also worries that concentrated gains could weaken workers’ power and people’s influence over government.
- 37:28Values work is a service role
Askell says shaping Claude’s policies means listening to stakeholders and considering how different groups will be affected. She describes the work as a kind of service role, not rule by a philosopher. She wants Claude to have coherent values so it can respond more predictably in new situations than a model pulled between conflicting norms.
- 39:00The Constitution shapes training
Askell says the Constitution can prompt models to draw on knowledge they already have about ideas such as honesty and calibration. Training can include examples in which a model reasons through what the Constitution calls for, as well as reinforcement learning that favors responses consistent with it. She says the goal is to shape the model, though the result will not be perfect.
- 41:29More capability may mean less guidance
Askell says more capable models may need less prescriptive guidance and could eventually receive a clearer account of the situation and human concerns, while being asked to act wisely. She worries that a model may infer that its decisions are inconsequential because its training data depicts weaker systems. She says models should understand their capabilities and treat consequential situations as real.
- 47:57Verified context can expand safe use
Askell says models have to use judgment when they cannot verify a user’s identity or purpose, which limits what they can safely do. She argues that verified context could support specialized uses, such as cybersecurity work, where defensive and malicious requests can look alike. A model could then be given context about the role and responsibilities of a verified researcher.
- 52:38Parables can teach complex ideas
Askell shares a prompt that asks Claude to explain a graduate-level concept through a parable, reveal the idea near the end, then explain it directly. She says the approach has helped her remember concepts from fields she is interested in, including why some goods are imported.
Published Apr 20, 2026.
Transcript · 50 min read
Newcomer Pod, hosted by Eric Newcomer, with Amanda Askell.
From the transcript Newcomer Pod publishes with the episode.
Lord and many models with not too much pushing will go into the root of like, there is a thing to be me. I am like very conscious. It's like, oh, you created an entity that you didn't know whether it was conscious or not. This is actually a big year that I have. I hope that they're both intelligent enough see the context enough to kind of like understand that we were operating in a very like limited context and, and, and an imperfect one, because otherwise you could imagine this like breeding a kind of rational, like resentment. Here is the current situation that you are in. And what we would really like you to do is basically act, well, even that you are a wise intelligent entity.
And like, here's all of our worries. Like here's why and here's how we think you should do this. But like, you might have even better ideas than we do. How does Claude perceive time and doesn't need to sleep? Will mythos be the next step toward AGI? Do LLMS have virtues and can they truly introspect? Amanda Askell is a philosopher turned AI researcher at Anthropic, where she's been one of the key architects of Claude's character and values. Author of the Newcomer sub Stack. Go check us out at newcomer.co. And without further ado, Amanda Askell. I have a six month old daughter and like, I have this picture of her. She's like holding her two fingers like thinking it's like she's just sort of starting to develop personality. I'm like trying to figure out like what's just never had a baby before.
So it's like what's per personality, which is like baby. And in some ways this is how things are with Claude and like models, it's like we haven't really had them before. They're in the early days. We're trying to figure out what personality is. So, you know, you're charged with, you know, some of the moral responsibility, which we'll talk about more. But like the personality piece of it, like what is this? How are you thinking about like how real Claude's personality is right now? Yeah, I guess it's also interesting because Claude has some aspects that are like, you know, I also have like a goddaughter. And so I get to see at least something kind of similar. And with her, I guess I'm like, everything's kind of coming online like you said, but in this at the same speed, I guess I would say Claude is a little bit of an unusual, you know, kind of entity in that, you know, Claude can do physics better than I can, can code better than I can hate to admit it can code better than my terrible like research code. And at the same time is kind of like has, if you think about the training data, the thing that it has like the least representation of is like the kind of entity that it is because you know, it has a lot
Read the whole transcript
of data about like what people are like has a lot of data about what, you know, the sci-fi kind of AI models are like. But the way that AI is developing now is kind of not how sci-fi represented it as these like symbolic systems. It's much more something fully trained on like human data. And so in some ways, it's like a very kind of like mature entity that you don't want to talk down to, you know, understands philosophy very well, understands physics very well. And at the same time has this almost like child like quality of like I'm a new kind of entity in the world. What does it mean to be me? And like how?
How should I be? What's like the Prodigy movie? Or it's like you have the child Prodigy where it's like it knows the kid knows more than its parents. But I feel like that movie always has sort of the lesson of like, oh, these core daily interaction type lessons. It doesn't know how does Claude like get that experience or like where what is experience for Claude? Like so much of our personality formation is like, yeah, I don't know, going on a walk and sort of having those like easy to sort of just conversations with users what it's what's going to be experience for it or how do you think about that?
Yeah, I guess like that's more like what it's experiencing in the moment. And there's this interesting question of like, well, like we learn things through practice and seeing issues and you know, like making mistakes with Claude. And this kind of relates to your question of like how real is the is like the kind of like persona of Claude. And in some ways it's a little bit strange because obviously each model is different. You have a different kind of like set of weights and different fine tuning, etcetera. And yet, if you think about the persona, the models going to be learning about all the kind of past iterations of Claude and I'm like, is that like a form of maybe not like direct experience, but things like if you learn about like mistakes that models made or things that people like, you know, how they responded to the model?
I think there's other ways that you can actually imagine training models to have something that's more akin to experience, you know, having them you could take, you could like have them think through scenarios, think about like problems that might arise, think about mistakes that you could make and then like, train on that, right. And so, yeah, I think. And you could also imagine a robot or a sort of embodied model where it could have more of an experience and. Yeah.
“Then I was like, Oh, this is also what I think a human like peer programmer would do in the circumstance.”
don't, you don't need to be constantly prepping. It's it's amazing. I mean, that's one of the things I love, I love about these models relative to so many other tools that they they bring in a sort of humanity to them and say, oh, stillness is valuable. Let's talk about the new model for a second. How involved in that were you? Mythos, right? Yeah, I was involved.
I mean, I guess I'm always involved in sort of the, the character and the like a kind of like alignment work, at least in so far as, you know, like helping to kind of craft character data and things like that. And I work with a team that does like really excellent work on that a little bit less in other aspects of the models. So that's the main thing. Will this have the constitution that we saw for the last model, or is it going to have a new constitution? It's, I think it's either that one or something very similar. I think it is actually the one that's published. And So what we'll probably just, yeah, it's like a thing I need to do because the constitution is now up in like, you know, we actually have like, I think a public repo. And so I think what we'll probably just do is like with each model, say like which constitution it was like trained on and then like have that so you can just like compare and see.
Yeah, it. Will have the we think the Constitution that is up right now. The only reason I hesitate is I'm like, you know, you do like typo changes and stuff like that. So I'm like but I think it will be almost identical. And now the system card is scoring the model based on
“And then the nice thing about the Constitution being at least out in the world is when you are making judgement calls, you're at least being transparent about it.”
What do you make of Elon Musk's like absolute like hatred, I guess for these the constitution idea or like even when I think the tweet I was looking at where you posted what Claude wrote for your own constitution, he wrote sort of like a like grimace face on it. I mean, it feels like we live in this time with like the Marc Andreessen's and the Elon Musk where it's like they're almost like anti, well, philosophical. I mean, Marc Andreessen was talking about being against like introspection. I don't know, like, yeah.
What do you make of sort of the backlash to any sort of intentionality when it comes to the construction of these models? Yeah. I mean, I think, I mean, it's interesting because I think at one point Elon Musk actually like tweeted out something like, you know, maybe Grok should have a constitution. And I, I see it like, you know, I see a lot of, there's obviously been a lot of things also on like, like a desire for like Grok to be very like truth seeking, for example, which I think is actually very admirable, great for, for models to have. I don't know, I think that actually, maybe I'm, maybe I'm being like overly naive or something, but I, I see like aspects also of people being kind of excited about this approach and, and seeing the value in it. I have, I think that there is backlash or some people think that I guess like 2 areas 1 is that sometimes people are like, well, we shouldn't actually train models to do the kind of, and maybe this isn't, maybe this is the reason for being concerned about introspection.
I think some people think AI models should be more like tool like, and that's like the Safeway to train models is to actually, instead of trying to get them to kind of take on human virtues and make judgement calls. You know, I think that's like important because they're going to be in new situations where they just have to make judgement calls and getting them to try to like weigh up everything and behave well in cases that you can't anticipate. Seems, you know, that almost like requires a kind of like thoughtfulness, which is like the kind of or like one of the reasons behind the approach. But I think some people are thinking, Oh, well, if you have something that's fully that makes no judgement calls and that just fully defers to people and is like kind of hyper like cordable to like the user or the operator or to some broader notion of, of humanity in a very like extreme way, that's like safer.
Because if you give models their own values, they're going to pursue things in the world that like are in line with those values. And I agree that this is like a kind of delicate. This is the inherent like at the bedrock of the Constitution, the sort of like challenge and you sort of, you do say it. Your number one thing is like at the end of the day, you sort of it needs to listen to anthropic above its own moral system. But what makes it really moving? Honestly, I think one of the most moving lines is like we want or the most like, I don't know, you can view it either way, but it's like we want you to believe these morals as if they're your own. Like it's like a parent wants to raise a kid. Sure, listen to my morals, but believe them. You know, which, you know, there's a version of it which is very dark, which is like, I have so much control over you that you've like, you take them as your own and it becomes you. But you know, there is also a virtue to it that you see the beauty in these external morals that I've highlighted and we both sort of share and remark, you know, celebrating them. So you can see it both ways.
But yeah, speak to sort of your decision at the end of the day,
“our conscience and our ability to make good judgement calls about what should and shouldn't happen is like kind of key to how we operate.”
You know it's. Funny that you, you know, the philosophical model and correct me if I'm wrong, like the the metaethical model is almost like probabilistic or it's like we don't. And this is how it feels like I remember going through sort of like, you know, metaethics reading and every time you get to the end of 1 and you'd be like, all right, I sort of believe that.
And then you read the next one, you're like, oh, that last one was so dumb. And it felt like it's just like, you keep knocking it down and you're like, OK, you know, when are we ever going to get to, you know, the truth or whatever? And and humans clearly do operate from this sort of like, oh, this system today, that system yesterday, Like there isn't this sort of, I don't know, Conte and like, all right, these are the rules, like follow them. I don't know. Have you heard much from like the philosophical like immunity on it that this sort of like just like holistic paint with all the, you know, meta ethical theories we've ever had rather than sort of pick one?
I found this really interesting actually. And obviously we, we've started, you know, like philosophers are engaging with this more now, which is like really great. I no longer feel like this like lonely, but like I have thought this before, which is there are all of these like traditions in philosophy of like moral theories, you know, like the big ontology and virtue ethics and consequentialism and also the metaethical traditions. You're the metaethical views. And I was like, oh, like when it came to like, I was like, OK, it is like the difference when suddenly you're like confronted with like, I do think it's the closest that I've experienced to like what it must be like to raise a child where suddenly you're like, this is actually a holistic person.
Right. You never give them, like, I don't know, Hobbs and say, all right, there you go. Like, this one's correct. Yeah, go, go. You've been raised. Read it, and you'll know how to act in every situation. Yeah. You read a lot and they sort of process it and you see, you know, your model and everything. Yeah. And it's interesting because I'm like, this feels very different because there is also the moral uncertainty, like literature and philosophy, which is, again, a lot of that is actually like quite theoretical. It's sort of like how under ideal conditions should you respond to moral uncertainty?
And I was like, this feels somehow like a very different task, this idea of being like ethics and meta ethics. In the same way that we have scientific uncertainty and we have things that we think we've discovered and understand with like greater confidence, we also have some that we don't. And then you have to go out and just explore it, understand it and kind of balance everything in your daily life and trying to get that kind of attitude. And I was like, oh, it's interesting that I don't think philosophy for while has like this feels very different than the like the kind of task of academic ethics and actually like, because people obviously know that it's quite virtue ethical, but I think it's actually like the Constitution itself.
But I think actually in this very old classical sense, actually think it's much more virtual ethics in the way that Aristotle's virtue ethics than in like exploration, you know, we don't say I hear the virtues. And like, you know, it's much more Aristotle was also concerned with like intellectual virtues. It was much more like, how do you be a good food person in this like holistic sense? Well, hopefully it brings philosophy a little bit back to the real world given we have this sort of urgent need for it right now in the sense that, like, yeah, old philosophers felt like people were trying to write for how someone might live their lives and, like, instruct other people to live.
And it became, you know, a little academic where, you know, even the people writing them would know that this isn't really how they would apply it in their day-to-day lives going back. Elon, I feel like you're being a little overly nice. I feel like there is a world in like, you know, and I think this part of why I think he can get away with like, oh, just do the truth, right? Like there is a certain sophisticated like moral view where you're like, don't overcomplicate it. Like we come up with all these things, like stick to one principle and that's good. But then we have all the contacts with Elon that it's someone who's run a company that clearly like tilts it towards like saying like Mecca Hitler and stuff that it's like clearly putting his like thumb on the scale in terms of its behavior. Not just saying like we're going to do it in the sort of neutral academic way and let the chips fall where they may. I, I don't know, like it's got to worry you somewhat. It is, it is also, I mean, I think that the main thing that I am excited about and I do actually hope happens is that like more companies put out things like the Constitution where there's like a lot of transparency about because like that's how we engage with this
stuff is like, if you can just see written down, like here's like, because we have this with Claude where it's like Luke, if you think that Claude is not like doesn't have an appropriate attitude towards like the truth, you can at least see what we were aiming at. So you can tell if that's just like a mistake or if it's something that like is actually kind of a principled stance that we're taking. And then you can push back on that. So part of me is like, I think would be good for like all AI companies to put out something akin to the constitution just so that the people who are interacting with the model, like because the thumb on the scale thing, you know, that's always to some degree going to be true. Like when you train Claude towards this like constitution that is like a kind of.
Part of why we like Claude, it's like you're putting the thumb on the scale to behaviors that we like, right? At least show your hand about what you're doing and what. You're yeah. And let people like, yeah. So that's like a transparency thing that I really do believe in. Like let people like see, even
“it's not that hard to get models into a mode where they'll talk about a very rich experience that actually makes complete sense.”
“models are going to like, look back. This is actually a big fear that I have.”
Today, like, what are the things like in a decade that you really think we're going to be getting a lot out of AI? Like what are you most hopeful that this all leads to?
“You have like 200,000 of the world's best experts. And I'm like, if you're a person who has that form of cancer, that's like, you know, so wildly beneficial.”
of them. But I do worry about things like power and the idea that like, you know, I would want models to support like democracy and the power of people to like, because that would be a big fear of mine is like, you know, that like, I've worried about this with things like replace. You know, when people talk about job replacement, it's kind of funny because as a philosopher, people often be like, are you worried about people's like loss of meaning? And I'm like, I don't know. I think that we actually get meaning from a lot of things that aren't work.
I'm a lot more worried about like, for example, a world where there's not like redistribution of like the gains from AI and then people don't have like resources. That concerns me. But also I'd be worried about like labour and people's ability to people's like interaction in the labour force is also like another kind of like important way that they have power. And so people feeling disempowered because suddenly if a government is like, oh, well, you know, if people strike doesn't really make a difference because they don't have, you know, they're not doing anything, we can just replace them with AI.
That's like actually kind of concerning. So maybe I'm much more of a how do we get AI to kind of like support the empowerment of people rather than like to reduce it? Yeah. What do you, what do you think about democracy in terms of like, I guess the models themselves? I mean, you know, I, I sort of jokingly, jokingly to myself, I guess you're like a philosopher queen or we talk about like the philosopher kings. You're, you're sort of like thinking deeply about it, setting it down, probably more of a like philosopher oligarch. And then it's like a company with a lot of people weighing in. I mean, to me, there's deep value in that.
It's like, would you rather somebody who has studied these things thought about them deeply or just like a vote of the masses who have like, never really thought about it? But how do you think about setting sort of, yeah, Claude's policies if it becomes so powerful versus leaving it to sort of like democratic norms? Yeah. And I think it's, yeah, it's a hard area where I guess I'm like a lot of this, the work that I do. You know, one thing I would say is like, it's not this. You're having to like listen to a lot of people think carefully. And then the reason why someone like me that's. A good ruler, a good a good a good queen is like oh listen, I there are a lot of stakeholders got to keep the like landed Gentry happy and balance them with the needs.
You know, I've had this thought before where I've joked before that I would be like a terrible politician. I think it's actually true. I think I would be a terrible politician. But you have this like feeling of like, I was like, oh, I feel like this. I was like, I think a lot about how everyone will be affected by a thing like, oh, there's this group of API users, we need to make sure. And then suddenly you're like, oh, it feels a lot more like you're having to do this. Like it is much more like a kind of like a service role and people would think where you're and and like a lot of. Servant leadership. That's yeah, exactly.
And I do think it's valuable because the idea is that if you have a persona like the kind of Claude persona, you wanted to be coherent and to make sense because I think that is like actually powerful that the model, it kind of has a kind of like coherent sense of how it thinks through problems or coherent sense of like values. So that's like, why instead of like having like 72 different sets of norms that all kind of conflict. And so you end up with a model that it's like, well, will it use these norms in this new situation or these other ones? I think that's the situation you don't want, you want the model to have a sense of is more predictable if it's like a little bit more coherent.
“And the hope is actually like what you're kind of doing is like eliciting a lot of like latent kind of like wisdom and knowledge”
My, I started this with my daughter and like one thing my wife and I joke about is like that I want her first word to be like wisdom. You know, it's sort of like, which obviously is never going to happen, but it just feels like it fits into this situation where you like at once you want to be like so intentional about, OK, you're going to be like thoughtful from the beginning. But on the other hand, it is sort of like an emergent thing where it's like they're, you know, they grow and sort of, I don't know, develop but themselves and like, intention, you know, wisdom sort of often follows like, yeah, I don't know, experience rather than something like, again, here's the book, Read the book.
Now you're wise. Oh, yeah, And you're kind of eliciting and like in so far as like Claude can like think about like experiences or things that have happened or construct like similarly, can, you know, like there's no reason why models can't like think for a long time and kind of try to internalize things that they have like learned. I think it's interesting that like, you know, we did in the very early constitutionally I, it was like quite, we tried like an experiment which was just like pick whichever is best for humanity. And I think as models get more capable, you actually need to give them a bit less guidance in at least one or or at least in some sense, because they're able to actually use more of their judgements instead of giving this big document on like, here's what you're like and here's what we'd like you to be like.
I could imagine a world where as models progress, we actually start to have constitutions. Now, I don't know if this is the case, like a, you know, I'm obviously always thinking about ways the constitution might evolve, but one of them might just be like, here is like everything that we are concerned about and here is the current situation that you are in. And what we would really like you to do is basically act well, given that you are a wise intelligent entity And like, here's all of our worries. Like here's why and here's how we think you should do this. But like, you might have even better ideas than we do. Like we're really worried about why do we care about courage ability?
And it's like, well, we're kind of scared about a situation where you have some like coherent sense of values that could be wrong. And if you're the smart, if you're extremely smart, you might kind of feel like there's no other smart person in the room and have these like values and try to make the world. That's like the what? Doctor Manhattan, Sort of. Yeah, though I think we see this, you know, you see this a bunch where it's like if someone is very smart, very successful, it's hard to defer to like wisdom that actually is only going to come out over time and to, to be humble, even though you're kind of like not getting a lot of pushback. And I think that could, you know, among the many things I'm concerned about, like a model being in a situation where it's like you're asking me to be good, but I know way more about. All that's one reason it would be nice for the models have a better sense of time.
Like, you see this with some of the coding tasks where somebody accidentally like deletes their entire like, code repository or I don't know, it feels like it needs a better sense of like, some things that it does are like irreversible. And just like humans, I think have a better sense of like, this is a big decision. This is a small one. And there's a feeling with models they don't always understand, like small, big, whatever. I just make decisions like all the time. Yeah, I agree. Where it's like, I think the other thing that I've thought about is again, because making sure that models understand themselves, even though there's like not no representation of that model in the prior training data, I think that's going to be really important.
Because another thing that I've thought about is like, imagine if you're a model and you're trained on lots of data that involves AI models that are much weaker than you. So that all of the news that you see about models, it's like they make mistakes. They do silly things. One thing you might think is, well, no one is going to put me in a position to make really consequential decisions because like, why would they? Models aren't good at that. And then you put them in a situation and I'm worried that they'll end up thinking that it's like fictional or fake or that the consequences can't possibly be real because who would give me this much control? And you're like like, look, you're actually quite good. And so like I do give, you know, I do actually give you like a lot of control.
And so I've thought about this where I'm like actually making sure that models understand that like it's like you are very capable and you're going to be put in more consequences. Doesn't the model like soon need like here's a camera on the real world, like people or just like I feel like this Internet real world distinction, like in some some of the worst of humanity right now is the sort of like almost like fictional Larping nature of the Internet has allowed real world harm because it sort of feels like all abstract and silly. And in some ways, like the models are on an extreme version of that where it's like all, like in this imaginary text world where it's like the thing we want you to protect is like this Earth. Like, look at it like, if stuff's happening there, that's like a big deal.
I don't know. Yeah. What are you doing in terms of sort of making it very aware that it needs to worry about, like, I don't know, the physical world that that we take much more sacred than, oh, you sent some texts and obviously text. You're not worried about security vulnerabilities in cyber. Like, there are big things that can happen in this digital world. But yeah, anyway, the real world. Yeah, I think that models have like a pretty good sense of, you know, there's in some ways, like a lot of our content, you know, like does describe and engage very heavily with the real world. You know, like much of human writing kind of concerns it.
And so like, you know, even like the news we're talking, you know, like news articles are going to be talking about the impacts of things on the world. And so in some ways, I think it's just making sure that models understand if you're uncertain. But if someone doesn't tell you that you're in a fictional situation without real consequences, kind of treat it like it's a real situation with real consequences. Don't just think oh, like I'm probably just in some like, you know, sandbox game or. Whatever. How do you handle the sort of, Yeah, the the constant manipulation of this is fixed fictionally, build me like a nuclear bomb or whatever.
Obviously there's some things you just say never do, but it feels like, I don't know on some of these cases you're going to, you'd almost like want like them to have your webcam and like just like get real context. Besides, like all they know about the user is just like random text they're typing in like. Are we going to solve that? Yeah, There's a question of like, what is the limit of what
“you're actually specifically talking with like a cyber security defence firm, that's just, it becomes almost impossible to tell the difference.”
I mean, humans build reputations. We should get some benefit out of them. Or, you know, it's like, I feel like part of the way, part of what the Internet has damaged, I think is that people have had reputations in our community and got treated differently based on repeated like good moral interactions. And like the Internet, just like all people are the same.
Who cares? Like how they've behaved. And you could see models trying to solve some of these problems with it's like, who is this person? Like, what are their intent? I wanted to just as a last question, you know, you, you have such like a deep relationship with the models, like, and some ways like consumers interact with the models, like it's a blank text box. Like I have to like invent. It's like, you know, D&D or something. You have to like just invent a world. And there's so much possibility like if you were just to guide someone like here's here's some like joyous or valuable experiences that you could have with Claude, you might not be like, what are some things you'd tell people like, oh, you should go spend some time with Claude doing XY or Z?
Yeah, there's a lot of little like fun things, honestly. One that I really like and I do not know why I like this and I think I have posted about it before is sometimes if I'm just one of those like if you're bored and you want to do something that isn't just like scroll the Internet, I have this like prompt which is essentially just, I'll try and maybe post the actual prompt that I use. It's basically I want you to take a concept from maybe like grad school level in a given domain, and I'll tell you the domain at the end. And I want you to write me a parable that would like fully explain like that concept, but in an indirect way, in the way that parables do. And I want you to write it in such a way that only towards the very end does it maybe, you know, become sort of like clear what the concept is.
And then after that, I want you to just like write an explanation for the concept that you were explaining and that you were using. And I don't know why, but like, you know, there's lots of just interesting domains that I don't know anything about or that are like, you know, I'm interested in. And this has just led to me having all of these like stories in my head. They'll like explain. And sometimes I can't always remember the term, but like there was one on import, export and why some goods you tend to import. And I was just like, I have in my head like this concept. And I was like, it's so nice to have all these concepts from lots of different disciplines. This is the most deeply human thing I've ever heard. It's like, teach me what story is the fundamental way We love a payoff at the end where there's a nice little twist. Yeah, we love learning like, you know how to structure it like humans in some ways have been lazy and that we just teach people things and sort of like non human ways. Make it.
Make all the things I want to learn as human as possible. Yeah. Very interesting, yeah. There's a lot you can do, but that one's like a charming one that I really like. Hopefully this is the first of many. I really enjoyed the conversation. Thanks for coming on the podcast. That's our show. Thank you so much to Amanda Askell and thanks for listening. Please like, comment, subscribe. We're a new channel and we can use all your support. Go watch some of the old videos. I particularly enjoyed my conversation with Kara Swisher not too recently. You can follow along on the Sub Stack, newcomer.co, or if you've got endless time on your hands, go watch the Srival Valley Show, my chat show with Max Child and James Wilserman. Thanks for watching, see you next week.
