Real Alpha
Try Pro free

Core Memory · 4 min · Dec 1, 2025

OpenAI avoids rewarding models for human-pleasing reasoning

Mark Chen · Chief Research Officer, OpenAI

Mark Chen says OpenAI chose not to supervise model thinking because human-pleasing reasoning could hide a model’s intentions.

In this episode

Full analysis
  1. 8:16OpenAI funds exploration to find the next research paradigm6 min
  2. 59:58Pre-training still has room to scalePro pick 4 min
  3. 1:09:16OpenAI wants AI to conduct research end to endPro pick 6 min
  4. 1:33:49OpenAI avoids rewarding models for human-pleasing reasoning4 min
  5. 47:17Chen rallied OpenAI researchers during the leadership crisis3 min
  6. 1:15:25OpenAI says it could use 10 times more compute3 min
  7. 23:49Models bring different intuitions to research problems4 min
  8. 0:00OpenAI focuses recruiting on the people it most needs4 min
“when you put incentives into the model to, you know, give you a thinking process that is appealing to a human, it won't necessarily be honest with you”
Mark Chenat 1:35:34 · Core Memory
1:35:34

Takeaways

  1. Incentives for human-pleasing reasoning could make models less honest.
  2. OpenAI chose not to supervise its models’ thinking process.
  3. Chen says observable reasoning can help researchers study alignment.

About this segment

Mark Chen says OpenAI chose not to supervise model thinking because human-pleasing reasoning could hide a model’s intentions. Keeping that channel observable also helps researchers study alignment.

Mark Chen

Mark Chen is chief research officer at OpenAI.

Led the teams behind DALL-E and Codex and visual perception in GPT-4, now oversees OpenAI's research.

  • Ashlee Vance, Host · CEO, Core Memory

    Wrote the best-selling biography Elon Musk; covered tech for Bloomberg Businessweek, The Economist and The New York Times.

From “How OpenAI Shapes Its Research And What's Next - EP 46 Mark Chen”, Dec 1, 2025. Watch the full episode

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

  1. 1:34:02

    first place? Yeah. So I think, um, really on the safety and alignment piece, um, I managed the alignment team at opening as well. And, um, I honestly feel like some of the grand challenges over the next one or two years are alignment. And I think for people paying attention to this slice of research broadly in the field, OpenAI, I think, has probably done the best work in the last year. And why I say that is like, there's been so much work on things like scheming, right?

  2. 1:34:28

    The more RL compute that you pump into the model, the more you can measure things like self-awareness, self-preservation, potentially even situations where the model can scheme. And it's scary because the model can come to you with the right answer at the end, the answer that you expect, but arrive at it from a very kind of twisted way right and um i think as the models do more complex tasks for us uh having a handle on what its thought process is um it's going to be super super important and okay chat you told me to ask you a question along these very lines which is i mean you're talking about a field mechanistic interoperability where we're trying to is a term that captures trying to understand this black box and how it operates. And I guess the heart of the question was, do our skills at doing that keep up with the complexity of the AI systems? Or do we just

  3. 1:35:27

    get to this runaway point where it's like, we're never going to learn how this thing works? Yeah. So I think one of the decisions that went all the way back to O1's release, which I'm very proud of is we decided that we weren't going to supervise the model thinking process. And I think when you put incentives into the model to, you know, give you a thinking process that is appealing to a human, it won't necessarily be honest with you, right?

  4. 1:35:58

    It won't say, kind of tell you it's true intention. And so we've actually, through that channel, been able to maintain observing the thinking process of the model as a tool towards understanding alignment. And there's a paper that was published just a couple of months ago with DeepMind, with Anthropic, really exploring kind of how this will evolve as a tool over time. And so I think we've made a lot of fairly good choices in design here.

  5. 1:36:28

    Yeah, I really do worry about this world in the future where the model will tell us something super convincing, but we can't be sure whether the model is aligned with us, right? Aligned with our values. And so I think there are a lot of interesting directions here. Like, can you set up games, right? Or can you set up frameworks or environments where, you know, like models supervise each other or they co-evolve together in a certain way, or like the only like stable equilibrium is one where, you know, the models are honest.

  6. 1:37:00

    And yeah, I think there's a lot of very exciting work to do there. Okay. All right. Okay. I'll behave myself now. Thank you so much for joining us. I am glad I'm old enough now that I don't have to take a job interview from like a super