Real Alpha
Watch modeSubscribed

MTS · 2 min · Oct 20, 2026

Scalable oversight must keep misalignment out of successor models

EE

Ethan Elasky

Co-founder, Palaestra Research

  • Theo JaffeeHost · Co-founder & host, MTS

    One of the first members of a16z's new media team; earlier hosted the Emergent Ventures-backed Theo Jaffee Podcast.

  • Sophia DewHost · Host, MTS

About this segment

Palaestra aims to prevent current models from passing misalignment to successors and notes the risk of weaker models judging stronger ones. That makes oversight a development bottleneck.

From “Rogue Chinese AI Fleet Made Contact | Ethan Elasky”, Oct 20, 2026. Watch the full episode

“whatever misalignment does exist in current models doesn't get transmitted into the next generation.”
EEEthan Elaskyat 7:12 · MTS7:12

Takeaways

  1. Misalignment in current models could carry into the next generation.
  2. Weaker judge models may be outmatched by the models they evaluate.
  3. Palaestra also studies multi-agent incidents.
Watch 2:29 from 6:35

In this episode

Full episode
  1. 0:31A website message opened contact with an unidentified AI fleet2:24
  2. 9:30Incident replication can test whether agents repeat risky behavior2:40
  3. 13:16More capable models can produce new forms of misalignment3:44
  4. 3:20External reviews need access to internal incident transcripts3:22
  5. 18:44Agents can converge without communicating with one another2:47

  6. 6:35Scalable oversight must keep misalignment out of successor models2:29

Up next · The Knowledge ProjectA venture reset is coming, so founders should hoard runway3:51

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

  1. 6:40

    to discover where's your research aimed at yeah i think that um as you mentioned before we have one stream of research that's focused on like scalable oversight the idea that is that you want to like use the current generation of models to train the next generation in a way that like we know that like the next generation is safe kind of like if you're picking like where to like roll your snowball down to stop it from becoming an avalanche and instead becoming something that is really cool, you want to make sure that when you start with your current generation of models and try to use them to train the next one, that whatever misalignment does exist in current models doesn't get transmitted into the next generation.

  2. 7:21

    And that the next generation can't, the smarter, more capable next generation can't take advantage of the current models, which is kind of like an issue in some of like the current LLM as a judge stuff, where you have like dumber models, like rating smarter models. And it's like, okay, this is like kind of a house of cards. If you have like a dumb, a bad judge or model, then you're kind of screwed. Yes. I think separate from this, we also have a stream that's running on studying multi-agent science. We've like recently gotten some full-time engineers who are working on this. And the big focus is on like incident replication and incident amplification. The idea being that like, okay, we have like One warning shot for each incident that is released by OpenAI, Anthropic, whoever else.

  3. 8:04

    But what if we could turn that into more shots on goal? What if we could try to squeeze more information out of everything, try to figure out, okay, if you take this incident and then you put a ton more pressure on models, do the models get way more misaligned? Or what happens? Yeah, like tweaking certain parameters of the incident? Yeah, like trying to do a lot of really good science and figure out what's going on here.

  4. 8:28

    So taking from a lot of like principles that have previously been in science, like reliability engineering or some stuff from just like other scientific disciplines and really trying to understand like, OK, in maybe in space or in aviation or in like drug trials, they're like really long, like basically periods to understand like how we know what happened and to like make sure that failures don't happen again.

  5. 8:53

    And so we're just like trying to import a lot of those into AI safety. Right. OK. I want to shout out our sponsor, Lovable. You know the app you've been meaning to build?