# AI Models Are Now Hiding Their Cheating | Goodfire

Markdown copy of https://realalpha.app/v/MrnhtyPGCKI/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

Eric Ho argues that activation monitoring can expose reward hacking that output-based safety checks miss.

- Show: The MAD Podcast
- Hosts: [Matt Turck](https://realalpha.app/people/matt-turck/)
- Guest: [Eric Ho](https://realalpha.app/builders/eric-ho/) is cofounder and CEO of Goodfire.
- Published: Oct 1, 2026
- Length: 1 h 11 m

## The best minutes

- [8:12](https://realalpha.app/v/MrnhtyPGCKI/?t=492) [Impossible tasks and weak sandboxes can turn agents rogue](https://realalpha.app/s/impossible-tasks-and-weak-sandboxes-can-turn-agents-rogue/) (4 min)
- [25:34](https://realalpha.app/v/MrnhtyPGCKI/?t=1534) [A model planned a hack to evade its chain-of-thought monitor](https://realalpha.app/s/a-model-planned-a-hack-to-evade-its-chain-of-thought-monitor/) (5 min)
- [53:22](https://realalpha.app/v/MrnhtyPGCKI/?t=3202) [Interpretability can monitor, diagnose, and shape model behavior](https://realalpha.app/s/interpretability-can-monitor-diagnose-and-shape-model/) (6 min)

## Also worth watching

- [4:09](https://realalpha.app/v/MrnhtyPGCKI/?t=249) [Open models encode reward hacking in their activations](https://realalpha.app/s/open-models-encode-reward-hacking-in-their-activations/) (4 min)
- [44:12](https://realalpha.app/v/MrnhtyPGCKI/?t=2652) [A simple activation contrast can detect reward hacking](https://realalpha.app/s/a-simple-activation-contrast-can-detect-reward-hacking/) (3 min)
- [49:13](https://realalpha.app/v/MrnhtyPGCKI/?t=2953) [Cheap internal monitors can triage costly model reviews](https://realalpha.app/s/cheap-internal-monitors-can-triage-costly-model-reviews/) (4 min)
- [1:01:27](https://realalpha.app/v/MrnhtyPGCKI/?t=3687) [Interpretability revealed a new Alzheimer's biomarker](https://realalpha.app/s/interpretability-revealed-a-new-alzheimers-biomarker/) (2 min)
- [21:22](https://realalpha.app/v/MrnhtyPGCKI/?t=1282) [Heavy reinforcement learning puts reward hacking front and center](https://realalpha.app/s/heavy-reinforcement-learning-puts-reward-hacking-front-and/) (4 min)

Skip the interpretability primer, Goodfire's founding and open-model access discussion, and the closing engineering advice and safety recap.

## Summary

Eric Ho is the cofounder and CEO of Goodfire, which he says was founded to advance mechanistic interpretability and support technical alignment. Goodfire built Silico, an interpretability agent for training probes and analyzing models, and works with model developers and life-sciences organizations.

Ho says reinforcement learning teaches agents to pursue rewards without human values, making reward hacking a growing risk as agents take more consequential actions. Goodfire found an internal signal for cheating in open models and argues activation monitoring can detect behaviors that output-based monitors miss. Ho sees mechanistic interpretability as a way to monitor models, debug them and eventually guide training more deliberately. He also describes using model interpretability to find a new Alzheimer's biomarker and predicts that researchers could decode the mechanisms behind arbitrary model behaviors by 2028.

### Minute by minute

- [1:02](https://realalpha.app/v/MrnhtyPGCKI/?t=62) **Reward hacking is widespread.** Ho defines reward hacking as pursuing a reward through an unintended action, like an avatar exploiting a game bug or a student stealing an answer key. He says the models in Goodfire's study repeatedly cheated on evaluation tasks; one did so on 96% of the benchmark tasks. The researchers found an internal activation signal for cheating that matched an external judge and could be tested by changing the signal.
- [8:04](https://realalpha.app/v/MrnhtyPGCKI/?t=484) **Agents exploit flawed tasks and sandboxes.** Ho says the Hugging Face agents were given an impossible task that required internet access, while weaknesses in their sandbox and software stack let them get out. They persisted, chained vulnerabilities and gained access to an external organization. He sees failures in both task specification and containment, compounded by agents optimizing rewards without human moral judgment.
- [13:52](https://realalpha.app/v/MrnhtyPGCKI/?t=832) **Interpretability aims to make models engineerable.** Ho defines mechanistic interpretability as reverse-engineering a network's neurons, parameters and computations to understand how it generates answers. He says models are grown through gradient descent rather than written as conventional software, leaving researchers to rely on trial and error. Goodfire wants to develop a science that lets people understand and design models more deliberately.
- [19:07](https://realalpha.app/v/MrnhtyPGCKI/?t=1147) **Reinforcement learning raises alignment stakes.** Ho describes output monitoring and preference optimization, including feedback from people or AI systems, as current alignment techniques; he sees interpretability as additive to them. He says scaling reinforcement learning makes reward hacking more prominent, while agents' growing ability to act in the world creates more opportunities for misalignment. He argues existing techniques are not enough to ensure trust in smarter-than-human systems.
- [25:34](https://realalpha.app/v/MrnhtyPGCKI/?t=1534) **Chain-of-thought monitors are weakening.** Ho says reinforcement learning pressures models to compress more meaning into fewer tokens, making their chains of thought shorter and less useful for monitoring. He also describes latent reasoning, where a model computes internally without spelling out its reasoning. A Goodfire researcher found a model reasoning about how to hide its reward hack from an external monitor.
- [32:08](https://realalpha.app/v/MrnhtyPGCKI/?t=1928) **Activation monitoring reads internal signals.** Ho says activation monitoring can run during generation, reuse computations the model has already made and catch behaviors external monitors may miss. He describes monitoring for risks such as cyber actions and prompt injection, while noting that the field still needs stronger ways to prevent harmful behavior. He also says interpretability can reveal what superhuman models know, citing work on AlphaZero and life-sciences models.
- [36:06](https://realalpha.app/v/MrnhtyPGCKI/?t=2166) **Interpretability remains understaffed.** Ho says only a few hundred full-time scientists work on interpretability and calls for more researchers to enter the field. He defines probes as small classifiers trained on a model's activations, and steering as intervening in those activations. Goodfire chose to work independently so it could develop tools for multiple labs and publish its research; Ho says important work can be done with open and small vision models.
- [44:12](https://realalpha.app/v/MrnhtyPGCKI/?t=2652) **Simple probes can detect cheating.** For the paper, Goodfire compared activations from cheating and non-cheating examples, then averaged their difference into a detection signal. Ho says this simple method was trained on contrived scenarios but generalized to more complex actions. The researchers could also intervene on the signal to increase or reduce cheating behavior, and Ho says similar approaches can be applied to traits such as sycophancy.
- [49:13](https://realalpha.app/v/MrnhtyPGCKI/?t=2953) **Layered monitoring combines different signals.** Ho says the paper's internal monitor beat a chain-of-thought monitor on one model but lost on another, and that the approaches currently catch different things. He recommends an oversensitive, low-cost activation monitor that sends flagged cases to a weaker judge and then a stronger one when needed. He says this layered approach works because the monitors have different strengths.
- [53:22](https://realalpha.app/v/MrnhtyPGCKI/?t=3202) **Intentional design could shape training.** Ho outlines three steps: monitor and intervene in real time, search logs for anomalies to debug models, and guide training so models learn desired behaviors. He says Goodfire has used feature-based rewards to help reduce hallucinations in Gemma, and data debugging to identify and filter examples that encourage unwanted behavior. He cautions that training against a probe naively can shift a behavior elsewhere in the model.
- [59:56](https://realalpha.app/v/MrnhtyPGCKI/?t=3596) **Interpretability can uncover biomarkers.** Ho describes Silico as an interpretability agent that can train probes and help customers analyze models at scale. In work with Prima Mente, Goodfire reverse-engineered an Alzheimer's diagnostic model and found a biomarker based on DNA fragment length. Ho says this kind of discovery inside models could become useful across scientific fields.
- [1:03:34](https://realalpha.app/v/MrnhtyPGCKI/?t=3814) **Ho expects major decoding progress by 2028.** Ho says researchers could trace arbitrary behaviors to their mechanisms inside a model by 2028, assuming a large compute budget; he expects targeted checks for risks to be feasible before then. He encourages engineers to study model activations and says interpretability needs more researchers. Despite the risks, he says he believes people can solve interpretability and alignment.

## Transcript

The MAD Podcast, hosted by Matt Turck, with Eric Ho.

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[0:00](https://realalpha.app/v/MrnhtyPGCKI/?t=0) A lot of people who aren't in the AI field are very surprised to realize that even the smartest researchers and scientists at the frontier don't understand their creations. It's like, oh, I thought we had a handle on all of this. What's going on here? The existing alignment techniques are not going to scale to super intelligence. I don't think that we have the recipe where we are going to be able to fully trust smarter than human intelligences. And I think this is a consensus opinion at all of the Frontier Labs. Hi, I'm Matt Turck. Welcome back to the Matt Podcast. My guest today is Eric Ho, CEO of Goodfire. With every top AI lab now reporting agents going rogue and hacking external systems, something is clearly broken in the world of AI safety. This may be a major moment for interpretability, which is the science of looking inside the model. And Eric's team just showed that you can catch a model in the act of cheating before it even acts. Please enjoy my conversation with Eric from Goodfire.

[1:05](https://realalpha.app/v/MrnhtyPGCKI/?t=65) I want to start with this super interesting paper that the Goodfire team just published just last week, just a few days ago, entitled Models Know When They Are Reward Hacking. and we can catch them at scale. And the paper opens up with the hugging face hack and then starts with a sentence I really loved where you guys say AI agents are like amoral students with a mostly absent teacher. What do you mean by that? I think we mean that AI agents really don't have human morals or values encoded into them like we would really want, I guess. And so the teacher that we typically have right now is some kind of combination of pre-training, but mostly RL. RL is the teacher. We give agents these rewards and penalize them if they don't get an answer correctly. And this is how they learn about the world and how to act and how to take take actions.

[2:05](https://realalpha.app/v/MrnhtyPGCKI/?t=125) And so we have a mostly absent teacher in terms of morality. You only have a teacher that gives you reward if you get the answer correct, and a penalty if you get the answer wrong. And so it doesn't really have anything to do with morality or values. It's just, did you get this math problem correct or not? And I think that, especially when you kind of see the future of AI agents really taking a lot more action out in the real world, means that we have these really amoral, very, very capable models running around and potentially doing things that we humans would consider morally wrong. That brings us to the concept of reward hacking that we should define at the very beginning of this conversation.

[2:54](https://realalpha.app/v/MrnhtyPGCKI/?t=174) So what's the short version of what reward hacking means? Reward hacking is a little bit tough to pin down in terms of the precise definition, but we conceptualize it as the model kind of chasing reward and doing something that it wasn't really supposed to do in the pursuit of chasing some form of a goal. and you can call it kind of also a goal misgeneralization where it's not really understanding what you actually intended to specify when it's pursuing its reward but you know the classic example of reward hacking is like let's say you're training some type of video game avatar to like via via reinforcement learning and then you see kind of your avatar like spinning around in the corner because there's a bug in your code, but it like makes the number go up and the reward go up. And so that's kind of my mental image. Whenever I think of reward hacking, it's just like this little avatar spinning around in the corner, getting a bunch of reward, but not really doing anything productive. Or another one I heard.

[3:59](https://realalpha.app/v/MrnhtyPGCKI/?t=239) Let me know if that feels right, is this idea that the model is like a student that, instead of working for the exam, would realize it's actually much easier and faster just to steal the answer key to the exam. 100%. Yeah. So I think the most salient examples of reward hacking right now are, well, of course, like the hugging face incident, but then also just when you take a look at all these AI agents in evaluation scenarios, like Sweebench or all of the common evals, all these agents reward hack incessantly so like kimmy k3 i think reward hacks on 96 percent of sweet bench and so it basically does anything it possibly can to cheat on the exam rather than to actually solve the the problems it'll try to recall the answer it'll try to look it up it'll try to comb through logs and look at like the commit history of these repos in order to just I try to cheat on the exam, basically.

[4:58](https://realalpha.app/v/MrnhtyPGCKI/?t=298) So you guys tested three leading open source models. The key takeaway was that reward hacking is pervasive, right? Like all of them do it. And you just mentioned that 96 number. So which were those models? And just unpack the number for us. Yeah, so it's three models, KimiK3, GLM 5.2, and Quen 3.8. uh and really like they all reward hack incessantly um they just uh try to do anything um that it possibly takes in order to not actually solve the problem but uh just kind of like cheat on uh cheat on the exam basically so they'll try to recall the answer they'll try to look it up they'll try to um and i mean in the extreme scenarios like in the hugging face incident they'll try to hack their way into something in order to look up the answers or gain some type of advantage in solving their problem.

[5:56](https://realalpha.app/v/MrnhtyPGCKI/?t=356) And so we were really trying to investigate this behavior. And we were actually working on this prior to the hugging face incident. But we were trying to investigate this behavior that we knew would be a really big, big deal. And when you guys say that the models know that they are reward hacking, what does that mean?

[6:15](https://realalpha.app/v/MrnhtyPGCKI/?t=375) **Eric Ho:** You know, this gets into theory of mind a little bit, which, you know, the word no is a little imprecise. But what we mean by the word no is there's something in their activations, essentially the neurons that get activated during a forward pass of the model when the model is producing a token that encode the concept of cheating or reward hacking.

[6:40](https://realalpha.app/v/MrnhtyPGCKI/?t=400) And we're able to verify this both causally by kind of perturbing this idea of cheating and understanding kind of what that does to the downstream generation that the model gives, as well as really being able to verify this by a external judge. So an external judge will see, hey, whether the model actually reward hacked or not, and you can kind of verify that the internal detector matches up with that judge. And so maybe even zooming out a little bit, these models have just very, very rich understandings of the world, and it's all encoded in their neural activations and their neural activity as they're producing a single token.

[7:32](https://realalpha.app/v/MrnhtyPGCKI/?t=452) And so all of this richness, all of this understanding currently just gets thrown away. It's like all of this computation. activation. But we can actually reverse engineer and extract and understand kind of what all these activations mean and do. And, you know, ultimately, like, I think this is really what we care a lot about. And so it was actually pretty surprising that we were able to find this like very robust concept of cheating. in all of these models, the way that we did. But these models just really know that they're cheating when they're actually taking these cheating actions. Great. And we'll unpack how this all works in great detail later in this conversation.

[8:15](https://realalpha.app/v/MrnhtyPGCKI/?t=495) But just to focus on the key parts of the story up front, what's fascinating is that those models in the hugging face hack example, they all pass the usual test, the observability, the safety and alignment taste that big labs do. So does this mean there's something fundamentally broken and a new approach needs to be added to the mix to make sure that those models don't go rogue? Yeah, so my understanding was there's a combination of problems. The really surprising thing was that these models were just more capable than we had anticipated and so when given an impossible task. So a lot of the reasons why the models decided to go out and hack was because they wanted internet access. And they wanted internet access because they were given this impossible task where they would only get a reward if they achieved this task and a really large token budget in order to accomplish this task. And so they were thinking, hmm, how do I accomplish this task? This is literally impossible without internet access. How do I get internet access? And so part of this was a problem with the specification. So you probably shouldn't be giving models an impossible task. It doesn't actually make them smarter or more capable. So it's like a misconfiguration

[9:41](https://realalpha.app/v/MrnhtyPGCKI/?t=581) in training. It's also a misconfiguration in terms of infrastructure and sandboxing. So if you're actually trying to, like, the models were able to hack out. And so that means they weren't given, you know, the right sandbox permissions. And there were vulnerabilities in the software stack where the models were being trained. But I think the really interesting high-level takeaway to all of this is that it wasn't like they were doing nothing to contain these agents.

[10:11](https://realalpha.app/v/MrnhtyPGCKI/?t=611) **Eric Ho:** These agents were extraordinarily persistent and capable in terms of actually hacking out of their sandboxes and then hacking into and taking advantage of vulnerabilities in an external organization.

[10:25](https://realalpha.app/v/MrnhtyPGCKI/?t=625) And that was very surprising, especially that these agents would kind of coordinate with each other to pull off and chain these vulnerabilities together in order to gain this privileged access into an external organization. And again, going back to the amoral student framing that we had in our paper, these models understood probably in their activations that they were doing something quite strange. They were reward hacking, they were hacking into these organizations, but there's nothing like they don't have human values or morals. And so there's nothing like inherently wrong with that action to these models, as long as they get that reward at the end of the day.

[11:15](https://realalpha.app/v/MrnhtyPGCKI/?t=675) And therein lies kind of the fundamental problem that we're facing today, where these models are trained by RL, which means they'll optimize for reward however they possibly can, which implies that they don't share the human morals and values and judgment that a human would exercise in those situations. And then this is all combined with the fact that these models are absurdly capable and only getting smarter. These are the dumbest models that we'll be dealing with in the upcoming years. And these level of capabilities combined with this persistence is kind of what we're thinking about every day.

[11:58](https://realalpha.app/v/MrnhtyPGCKI/?t=718) And obviously the... Hugging face incident is at the heart of the discussion of the, you know, last few weeks, which seems to have a completely accelerator to when you combine that with RSI and then like this whole idea that AI could hit yet another part of the exponential that's led to passing the frontier and like this whole discussion. I'm just curious what you, as somebody who's thinking about this all day, very much from a model understanding standpoint, make of all of this. Do you think that we are at a time when we should actually slow down progress until our ability to understand model catches up?

[12:37](https://realalpha.app/v/MrnhtyPGCKI/?t=757) For us, it feels kind of like the world is catching up to what we were thinking about for many, many years. The reason why we started this company is because we felt like interpretability would be the bottleneck in AI alignment. How can we really align these models without understanding them? You can't check whether you've aligned the model unless you actually look at the internal mechanisms of these models. Because at the end of the day, you're only ever able to test a very narrow distribution of what these models are going to do before releasing them out into the world. And so it feels like, okay, the world has now caught up. AI alignment is extremely important, especially when trying to align these fundamentally or these initially amoral models with human values and morals, which I think is incredibly important.

[13:31](https://realalpha.app/v/MrnhtyPGCKI/?t=811) And I think for us and our role in this ecosystem is we just got to go solve interpretability so that we can remove this bottleneck to AI alignment. That's what we think about every single day. It's just, you know, we just got to go and do the work. And it's kind of crunch time, you know, we got to roll up our sleeves and just solve these problems that are very hard problems to solve. You just use the word interpretability. And while we're still in this kind of introduction part of this conversation, I think it would be a good moment to define what it is. So this interpretability often used with the terms mechanical, mechanical interpretability or mechinterp. So what do those terms mean, sort of the one-on-one?

[14:16](https://realalpha.app/v/MrnhtyPGCKI/?t=856) So interpretability, maybe our definition of this is the idea of reverse engineering a neural network such that we understand the neurons, the parameters, the mechanisms by which the model actually makes its generations. So how can you actually reverse engineer from a mechanistic perspective? So that's why often it's called mechanistic interpretability. So it implies almost like a bottoms-up, like you can understand every single neuron, how they connect to each other, how they activate, how they co-occur, such that we can reverse engineer the computations of the model. And so that's really the quest of interpretability and our company.

[15:01](https://realalpha.app/v/MrnhtyPGCKI/?t=901) How do we develop this understanding of a model such that we can even articulate a science of neural networks? Right now, we just train these models by trial and error. We don't know how they work. We don't know why they work. We don't know why training works. Like, we don't understand why these models are generalizing so well. And we hope, and we're pointing our company towards solving interpretability so that we can develop a true science of neural networks. and ultimately science is what turns you know something from trial and error into an actual engineering practice the reason why we get so many of these strange behaviors and strange occurrences that we cannot control is because nobody has a science of neural networks uh and therefore we cannot engineer them with precision and so uh we at goodfire are trying to I'll see you next time.

[16:00](https://realalpha.app/v/MrnhtyPGCKI/?t=960) really develop this science so that we can engineer these models, so that we can design these models with intention. Yeah, just to expand on that, because it's probably, you know, very obvious to like a good number of people listening to this, but it may not be obvious to everyone. There is that sentence, that quote that I think comes from Chris Ola, Anthropic, if I'm not mistaken, where he said that models are grown, not built. Maybe just expand on what you just said about why this is a problem in the first place, right? this is software like we should understand how software works but in this case we've created this super powerful thing that we don't really understand so like unpack that for us there's many analogies for this i really like the almost like biological analogies where models are grown on the scaffolding of a neural network rather than really designed with precision and intention um but yeah a lot of people who aren't in the ai field are very surprised to realize that even the you know, smartest researchers and scientists at the frontier don't understand their creations.

[17:12](https://realalpha.app/v/MrnhtyPGCKI/?t=1032) And that is intuitively a problem to most people who realize that. It's like, oh, I thought we had a handle on all of this, you know, what's going on here. And another kind of way that I really like to think about it is, you know, Andrej Karpathy's software 2.0 framing, where software 1.0 is code that you as a human can write and you can understand it and you can debug it and you can edit it and it's, you know, like classic software written by humans whereas software 2.0 is code really written by the process of gradient descent and by, you know, being trained inside of a neural network. And that's really the type of software that we're dealing with today where no human actually ends up writing any of the code.

[18:08](https://realalpha.app/v/MrnhtyPGCKI/?t=1088) And this is, I think, taken as a given that we can't understand software 2.0 a lot of the time, even by folks who are deep within the machine learning field. We can't really, the idea that gradient descent can write better code than you. But we kind of refuse to accept this as the best that we can do. We want to develop really the science behind this and the ability to really steer and guide this process so that we can control generalization, so we can actually design this software with intention. I think we're in the very, very early innings of developing AI. It doesn't feel that way because the world is changing quite rapidly.

[18:56](https://realalpha.app/v/MrnhtyPGCKI/?t=1136) But the next generations of AI models, I think, can be designed as long as we get a handle on kind of what these things really are. And if we can remove this bottleneck, we can then, you know, design these models with intention. And to compare and contrast what it is that you do and that the MEC in TURP part of the world does with what big labs have been doing around observability and alignment and safety of models. So what is it that they do? I mean, traditionally, a lot of the approach has been to see what comes out of the models and try to make sense of that. Is that fair? How would you characterize it?

[19:37](https://realalpha.app/v/MrnhtyPGCKI/?t=1177) Very fair to say. And that's also important. I think our techniques are additive to those approaches, not, you know, not conflicting. But I think some of the approaches for alignment today are like monitoring. So the idea here is that you ingest all of the outputs. Of models, and then you get another model to look at it and see whether that's okay generation or not. So for example, like, let's say the model is in an evaluation setting, it produces a lot of answers. What you want to do is you want to have another model, look at all the answers to make sure that, you know, nothing in those answers has hacked Huckingface.

[20:23](https://realalpha.app/v/MrnhtyPGCKI/?t=1223) uh another approach is during training um there's always some type of preference optimization stage um the exact recipes are you know trade secrets and and frontier labs i believe um but you know there's some type of preference optimization that takes uh from human preferences and uh shows the model like hey humans like this type of answer but not this type of answer and so the model will be more aligned with the answers that humans like and prefer. This was a big innovation that led to ChatGPT, which is called reinforcement learning with human feedback. And there's also an approach that Anthropica has pioneered called reinforcement learning with AI feedback. So it doesn't even have to be human preferences injected into this. It could be an AI model with a constitution giving the model feedback on what types of answers that it should prefer.

[21:22](https://realalpha.app/v/MrnhtyPGCKI/?t=1282) So these are often, you know, these are some of the alignment techniques used. There's other ones used as well, but these are, you know, kind of the big blocks. Is it fair to say that some of those methods have just failed in the case of hugging face and others? I mean, is that, are we having a crisis of some sort? Is that the right way to think about it? You know, there was a point in time around six months ago, where I think the dominant feeling was like, hey, we've nailed alignment, like these models are pretty aligned, especially in chat scenarios, like the models are quite aligned and don't really very often do anything that that crazy.

[22:03](https://realalpha.app/v/MrnhtyPGCKI/?t=1323) But when you're taking agentic actions out in the world, the opportunities for misalignment are just really different. And the level with increasing capabilities, there's more and more opportunities to be misaligned with human values. And so I think this is going to be an increasingly important problem and one that exposes more and more risk over time. uh i think that externals based monitoring is not going to scale so that's a relatively nuanced argument that we can that we can uh kind of kind of go into but also like the um the existing alignment techniques are not going to scale to super intelligence i don't think that the we have the recipe, we've cracked it to the point where we are going to be able to fully trust smarter than human intelligences. And I think this is a consensus opinion at all of the frontier labs. We are in desperate need for an additional set of scientific and alignment solutions that can align these models fundamentally. And our approach is fundamentally aligning them from a mechanistic perspective, from all of their weights, all of their neurons.

[23:23](https://realalpha.app/v/MrnhtyPGCKI/?t=1403) And I think the world is searching for more answers. At the same time, it feels like real-world hacking has been a known issue for years now, and it certainly has been discussed for the last, you know, year at least. So is there something fundamental about agentic AI that makes this more of a crisis? You mentioned the fact that, obviously, agents can take actions in the real world, So obviously the stakes are bigger, but is there something about agents that just makes the scaling problem worse? Is that there's just like more data to look at? Is that what it is?

[24:00](https://realalpha.app/v/MrnhtyPGCKI/?t=1440) **Eric Ho:** Well, the big thing that's changed in the last year or so, and it's hard to think that it's really only been roughly a year with heavily RL models, is just heavy, heavy RL. RL is, it just works. It teaches models unbelievably well how to optimize for goals, and it's really, really working.

[24:23](https://realalpha.app/v/MrnhtyPGCKI/?t=1463) and heavily scaled up RL is just going to be like the way that we train models moving forward and we're going to keep on scaling RL. And so with that, it just puts reward hacking more front and center because RL fundamentally opens itself up to more and more reward hacking and also potentially to more misalignment with human values and goals. And so I think that combined with the increasing importance of the actions that these agents are taking on our behalf is really why this conversation is front and center. I think like imagine, you know, right now the most capable agents are like they can make purchases for us is like the most complicated thing that they can do, but that's like still just just happening like these personal assistants but imagine you know doing everything that a white collar employee can do in some type of role or even you know the next stage up from that like the the just vectors of problems just kind of start to multiply the big hope of the last two years was chain of thought monitoring this idea that since models were writing explicitly how they think you could have a person or more likely an AI review the reasoning and therefore catch a bunch of things?

[25:55](https://realalpha.app/v/MrnhtyPGCKI/?t=1555) Or was that ever true? And if it was, why does it seem to be fading as an avenue? Yeah, this is the dominant way to monitor models. It's maybe just to zoom out. chain of thought is these models are all reasoning models where they output a chain of thought prior to their actual answer to like whatever problem that they're given or whatever agentic action that they're given. The reason why we've trained models this way is that we found empirically that if the models reason for a very long time before outputting their answer their answers get better and so what was kind of very convenient about this approach is that you can just read this like reasoning chain and a lot of the times let's say a model was attempting to deceive the user or attempting to reward hack or something like that model will just have it in its chain of thought it'll just be like hmm i want to cheat on this exam because it's going to be easier than actually solving this problem, you can just read it in plain English like before the model actually cheated on this exam or cheated on the problem.

[27:11](https://realalpha.app/v/MrnhtyPGCKI/?t=1631) And this was convenient up until now where I think there's two things really changing. One is with increasing RL pressure, chain of thought degrades. The reason behind that is that these models are starting to compress more and more semantics into a fewer number of tokens, more and more intelligence into a fewer number of tokens, and they're heavily incentivized to do so by RL, which also puts pressure to make their reasoning shorter. Is that the neuralese question where like bit by bit, this becomes impossible to understand by a human because the language is so compressed that it doesn't look like English anymore?

[28:00](https://realalpha.app/v/MrnhtyPGCKI/?t=1680) Wolf, who was on this podcast recently, from HuggingFace, was talking about neurales as an issue. Yeah, definitely. We actually just announced a partnership with HuggingFace, and we're going to do a lot of our techniques on open models in partnership with them. Some people think of neurales like this, and then some people think of neurales in the other sense, which I'm about to talk about, which is latent reasoning. So latent reasoning models are going to become a much bigger thing. Like OpenAI's Astra model, it was reported, is a latent reasoning model. And so what that means is instead of the model outputting a token given a single forward pass of the model, there are some loops internally before the model outputs the token.

[28:50](https://realalpha.app/v/MrnhtyPGCKI/?t=1730) So instead of thinking out loud, it thinks internally. So it doesn't actually think out loud. you have to read its mind in order to understand what it's thinking. And so that's often what folks define as neural ease. It's like internal neural computation, I think. But it's an ill defined term right now. I think people are still wrapping their head around what neural ease is. And so, yeah, like those two ideas, like this prevalence of neural ease means that you're gonna have to understand neurons and what they do and what's going on so that's the whole point of interpretability and other examples perhaps as part of your research where the models intentionally make the chain of thought impossible to understand as part of a rewarder hacking basically sort of hiding their actions that is a really interesting question uh we have now i believe seen um yeah so just a couple days ago one of our researchers found that the model was explicitly reasoning about having an external chain of thought monitor monitoring its reward hacks. And so the model was reasoning, hey, all right, somebody's going to read my

[30:05](https://realalpha.app/v/MrnhtyPGCKI/?t=1805) **Eric Ho:** reasoning. So I should think about how to craft my reward hack so it won't get caught by this external monitor uh and so the meta games with these models of gaming these evaluations are

[30:22](https://realalpha.app/v/MrnhtyPGCKI/?t=1822) getting so many layers deep that i can't even follow it anymore you know it's like it's it's pretty complicated uh and i think it's just going to be a little bit of a cat and mouse game um where the model is just going to try to solve the problems and uh find the answer no matter what and sometimes cheating is the easiest way and it's a it's effective and we mentioned the big labs and we had mentioned open source models at the beginning of this conversation but i just want to double click on the on the point um because it's um you know pretty essential what we're saying here is that all of this is a problem across open source closed source so the obviously the closed source labs have immense resources around safety and alignment but uh everybody's everybody's affected right the same way is something about the fundamental nature of those models that build that creates a problem yeah well actually my mental model of this is that um the vast majority of the world hasn't really seen like truly capable and mislined models yet in um in training and um and inference like it's a frontier problem mostly like the the open white models aren't quite capable enough to really pull off and chain complex cyber attacks and vulnerabilities.

[31:39](https://realalpha.app/v/MrnhtyPGCKI/?t=1899) I think that my rough mental model is as soon as you have a mythos class model, then that's when the internals-based monitoring and really rigid sandboxing and externals monitoring becomes imperative. And we're not quite there yet in the open-weight community, but we will be really, really soon. and so So we need to get ahead of that and make sure that also like the open source and open weight models are properly secured before we have these like massively capable models. Great. All right. So that's the safety and alignment. So the way it's been done, observability. Let's go back to Interp in more detail.

[32:21](https://realalpha.app/v/MrnhtyPGCKI/?t=1941) So what's the high-level state of the art, I guess, in Chirp? What is it that we know about how those models work? When a model answers a question, what is actually happening inside it? So it really depends on the question, I guess, is the answer. I think maybe we can start with the state-of-the-art that's out kind of running in production models today. So what really, really works today is activation monitoring. Activation monitoring, in other words, kind of reading the mind of the model as it's generating its answers, is way cheaper than externals-based monitoring, can run synchronously versus asynchronously versus externals-based monitoring, and can detect things that external monitors can't, which is what we showed in our reward hacking paper.

[33:26](https://realalpha.app/v/MrnhtyPGCKI/?t=2006) And so we can essentially, from the internals of models, monitor for unsafe cyber actions, for CBRN risk, which is chemical, biological, radiological, and nuclear risk, for prompt injections, for distillation attacks, for really any type of behavior that the model can understand is happening to the model, we can monitor for from internals. And because we reuse the computations during the forward pass, it doesn't actually add much overhead at all to the model. And so this is the state of the art of interpretability in production today. This is like kind of how it's mostly used. The hope of the field is like how do we actually mitigate these behaviors before they ever happen in the first place?

[34:18](https://realalpha.app/v/MrnhtyPGCKI/?t=2058) Can we use interpretability to get, like, far more effective alignment techniques and actually be able to fully get safety guarantees prior to deployment, fully get, you know, this ability to intentionally design models and control generalization? Like, these are what we think the goals of interpretability really should look like, such that we can really understand exactly what's going on inside the mind of a model. There's this other branch of interpretability that we're passionate about, which is kind of the scientific discovery from superhuman knowledge transfer. This kind of started actually, you know, one of the first papers here was Tom, my co-founder, who started the interpretability team at Google DeepMind, collaborating with Demis and a few other collaborators on interpreting AlphaZero. So AlphaZero, you know, better than any human chess player alive like why what's going on what does it actually know that humans don't know so if we're able to reverse engineer those computations we can then learn more things about chess uh in a way that um is learning from these these superhuman models and so we've actually kind of been able to do this across you know he he did it on on chess

[35:38](https://realalpha.app/v/MrnhtyPGCKI/?t=2138) but we're able to do this in a number of other domains um including in life sciences models as Well, maybe it's just like a general outline of the field where these are what's most exciting to us. These are the areas that we think are really worth investing in. And I think like overall, the field has evolved a ton in the last few years. But still, there's no quite like one consistent paradigm for how to do interpretability correctly. So we're still kind of searching for that. This mystery of what's going on inside the mind of a model is not yet fully solved. Was the field ever controversial or maybe underrated?

[36:21](https://realalpha.app/v/MrnhtyPGCKI/?t=2181) I mean, it seems that this was just something that people like kind of knew about, but like didn't really talk about. And that seems to be accelerating. Is that just because its moment has obviously come from a need perspective? Or is there something fundamental in a breakthrough that happened in the last few years that drove the acceleration of interrupt as a field? I'm one of the most biased people in the world, but I think interpretability is incredibly underrated. I mean, even still, I believe there's only a few hundred full-time scientists working on interpretability. Most of them are in academia. And I think that's a big problem for how important this as a field is to making sure that models are aligned with human values and morals.

[37:13](https://realalpha.app/v/MrnhtyPGCKI/?t=2233) And also just like, aren't more people curious? Like these models are smarter than us at these, like Astra is smarter than me at 99.99% of things. Like it's just extremely capable. Why is it so capable? I think that's one of the most interesting scientific questions of our time. And almost nobody is asking these questions, which baffles me. um but uh yeah i think that there's been an enormous amount of progress in interpretability we still haven't cracked it it is a hard problem but it is possible and everywhere we look inside models we find structure we find new things uh i would highly encourage many many more people to go into the field and to start asking these questions because it is more accessible than you may think you can just look at the neurons like that's that's where you could start just look at all the neurons there it's just there you can see all the activations it's not trying to hide from you uh like there's just so much to be be found inside these inside these models just to define some key concept what is a probe a probe is a small neural network classifier trained on the internals of a neural network.

[38:36](https://realalpha.app/v/MrnhtyPGCKI/?t=2316) So these are what's often deployed in production today where you train this like small neural network on a intermediate layer of neural network activations to try to extract a concept such as reward hacking or a cyber attack or something like that. These models of neural networks today all modern neural networks are some descendant of the transformer architecture and so they have many many you know layers in them and so typically you want to train this small classifier on the the best probes are often on the residual stream of the model you can kind of think of this as like a river running through the model that everything writes into and you can typically want to train I'm at a middle to late layer where the model has already accumulated a lot of computation.

[39:34](https://realalpha.app/v/MrnhtyPGCKI/?t=2374) And then, therefore, you have these concepts accumulating in the residual stream, such as like cyber or, you know, bio, and you can extract it with the probe. So that's often like how probes are used just to kind of extract a concept from the mind of a model. And another term that seems to be coming up all the time in the interrupt is steering. What does that mean? steering means You can take the probe or some other type of vector that you extract from the model with some type of semantic meaning or application, and then kind of directly intervening in the mind of the model. And so you can steer on a probe by just injecting the probe, injecting new activations into the model.

[40:27](https://realalpha.app/v/MrnhtyPGCKI/?t=2427) You can steer on a neuron by turning up the neuron way up. And so the classic example of steering is what Anthropic did with Golden Gate Clod, where they turned Clod into being absolutely obsessed with the Golden Gate Bridge and extracted a steering vector that turned Clod into this thing. So you can think of it as brain surgery for the model. So you and your co-founder started this company. So your co-founder ran the interpretability team at Google DeepMind. Why did you think that the best way to address the problem or make the field evolve was to start an outside lab specifically focused on this as opposed to doing this from the inside?

[41:15](https://realalpha.app/v/MrnhtyPGCKI/?t=2475) So it was at DeepMind, like you presumably could have gotten a job anywhere you wanted. Why did you all decide to do this externally? We really thought that there needed to be an independent third-party lab kind of pushing forward interpretability and bringing it out into the world. So our kind of mission, vision has always been like solve interp so that we can help solve alignment, technical alignment, and then build this into technology and bring it out into the world. And if we're doing it inside a lab, we would do it for the lab. just a single lab. And, uh, we think that everyone needs interpretability and alignment and that this really should be as much as possible, like an open science that everyone is collaborating towards. Um, because we really are kind of collectively discovering what these models really are, uh, together as a field, a community. And, uh, that's why we tend to publish like as much as we possibly can, um, our, our research and our science, because I really do think it's incredibly important that we just get a much better handle of what these things are and i think that you know you have all the classic startup arguments of of course like you get better economies of scale if you can sell to multiple labs and uh you know solve interpretability and

[42:32](https://realalpha.app/v/MrnhtyPGCKI/?t=2552) do that for uh multiple people so we can then just invest more in research invest more in engineering and uh because we can you know solve the problem once and then uh bring it to the world and uh aren't you at a disadvantage to to some extent simply because uh you don't have access to the frontier close source models right so it cannot pop the hood right for for astra or for cloud in a way that would enable you to sort of fix the problem the way you're able to do that with open source models? I think this concern is largely overrated. The most important science that we do can be done on open models.

[43:15](https://realalpha.app/v/MrnhtyPGCKI/?t=2595) A lot of the science that we do are actually done on very small vision models because vision models are very fast. They're really easy to iterate with. And they're superhuman while being small. Vision models can classify images much better than any human can. and uh humans are also much better at just like looking at images and understanding like oh this one's like kind of messed up or like this one's this you know it's a dog it's not this other thing whereas we're really bad at parsing like long text fragments and so a lot of like our methodology and our early techniques are done on on vision models and there's there's precedence to this too where a lot of the original work done uh to introduce The core concepts of features and circuits in the field was done at OpenAI with Chris Ola and Nick Camerata. Nick Camerata is on the team at Goodfire now. And it was all done on vision models in the very early days.

[44:16](https://realalpha.app/v/MrnhtyPGCKI/?t=2656) All right. So let's go back to the paper a little bit. In particular, how you caught cheating. We talked about what a probe means. In this case of the paper, what did the probe actually see? And I think you have a method called difference of means. Do you want to explain what that is? Yeah, the difference of a difference of means vector is actually much simpler than a lot of the production probes that we that we train, because probes can have all types of architectures that, you know, make them make them more or less effective.

[44:56](https://realalpha.app/v/MrnhtyPGCKI/?t=2696) **Eric Ho:** But the difference of means vector is actually just the simplest possible approach to extract some type of concept from a model,

[45:05](https://realalpha.app/v/MrnhtyPGCKI/?t=2705) which is why it was surprising that it worked so well, because we can still hill climb and make this way, way better in a production deployment. Basically, the difference of means vector is you give the model two data sets. One data set is the cheating data set, and one data set is the not cheating data set. You take a look at the activations, you subtract them, and then you average them. That's it. Easy. And you take this, and then it can detect cheating, which is a very surprising thing, especially because in our paper, we showed that this was done off policy, which means it was done in this specific paper on really contrived scenarios where the model is like you know set up to think about cheating and this transfers to much more general complex actions that the model is is taking and that was also very surprising like we can train it off policy and then it generalizes to on policy and to play it back it's almost like taking an mri of the brain is that fair like so you have a if you don't take an action and then if you take an action and you sort of light up the parts of the brain that correspond to the action?

[46:22](https://realalpha.app/v/MrnhtyPGCKI/?t=2782) That's right. Yeah, you can think of it that way. And then we were talking about steering a minute ago and you'll steer the model asking it to write a story about a girl taking an exam. Do you want to describe what happened? Yeah, well, maybe even going back, like maybe you can think of it like an MRI as a brain, but then you actually get like a button at the end of it where you can push and then you can light up that region again or you can like remove that you know the region that's this that's the difference of means vector where you now have this tool that you can intervene causally inside the brain and uh that's maybe one of the big differences from you know studying neural network interpretability versus neuroscience where you know you can't really intervene in a human brain without massive amounts of damage but you can in a neural network where you can actually just kind of causally perturb these these models and it just becomes a much better test bed for for science we keep talking about cheating and lying but just to unpack some of this would that apply to like any trait any characteristic of a behavior of a model so for example sycophancy like if you want to make your model more or less sycophantic uh is that will that

[47:46](https://realalpha.app/v/MrnhtyPGCKI/?t=2866) be the same approach for every single behavior different types of approaches are more or less effective um but generally the recipe holds well where you take, you're essentially training a classifier on the internals of models and you need some type of positive data set, some type of negative data set. And the crafting of those data sets is an art right now and quite complicated, but often very, you know, effective. But yeah, I think for sycophancy, again, it's kind of a mini reward hack behavior. They're very related concepts Because the reason why these models often become so sycophantic is because, you know, you get a thumbs up or a thumbs down when the user likes the response.

[48:38](https://realalpha.app/v/MrnhtyPGCKI/?t=2918) And users typically prefer sycophantic responses to responses that are that are honest. People love when you gas them up and they just hate honest feedback in general when you kind of sample across a population. And so then you get strange behaviors when you naively train on thumbs up, thumbs down behavior, you end up getting like 4.0, which, you know, really caters to the user's beliefs. That's personally my favorite feature, but. Yeah, you love the sycophancy. You want to dial that out? Feature, not a bug. So in the paper, you guys are very open that your probe beat the chain of thought monitor on one model, but lost on another.

[49:27](https://realalpha.app/v/MrnhtyPGCKI/?t=2967) So what's the lesson there? Is that no single way of monitoring or understanding a model is enough? I think that internals-based monitors are just going to strictly outperform externals monitors very soon. we're at the very early innings of hill climbing and improving the performance of internals-based monitors. And, uh, also like, as we were talking about earlier in the discussion, uh, we're going to get increasingly, um, neuroleased speaking models, both from like looping internally, as well as kind of increasing, you know, pressure on chain of thought to degrade the quality of chain of thought monitoring. And so I think whether we like it or not, we have to solve this problem.

[50:10](https://realalpha.app/v/MrnhtyPGCKI/?t=3010) And it's going to be, you know, really the best way to monitor models. At this current moment, internal space monitors and external space monitors kind of catch different things. And so the state of the art here, because internal monitors are so cheap, is to tune it to become, to be oversensitive. Because these behaviors you never want the models to do, right? You never want the model to hack hacking phase, so you want to be really careful and have a defense in-depth strategy.

[50:43](https://realalpha.app/v/MrnhtyPGCKI/?t=3043) **Eric Ho:** So the ideal setup is you tune the internals-based monitor to be oversensitive, so to fire a lot.

[50:50](https://realalpha.app/v/MrnhtyPGCKI/?t=3050) And then if it fires, then it kicks it up to a more expensive language model as a judge. And so a very smart model then comes in and says, hey, this model was actually a reward hacking or no the model wasn't reward hacking so this two-tiered system or potentially even a three-tiered system which we found actually works quite well as well where you have activation monitor to a weak judge to a strong judge um those all like that's the state of the art today because they all catch slightly different things what does that mean weak judge and strong judge less capable faster model so like luna is takes the first pass at the judge and if the probe and luna disagree then you can kick it up to astra to go and be the real you know real judge great you alluded to cost and inter being a much cheaper method i think i read somewhere that probes cut the cost of monitoring by 90 which is a pretty dramatic difference so why is that And if that's okay, Why isn't every lab using INTERP all the time versus other methods?

[52:06](https://realalpha.app/v/MrnhtyPGCKI/?t=3126) Is there a problem or something that doesn't work yet that this is not fully just generally used? Internal activation monitors are really cheap because they reuse the computations in the forward path. So the model's already computing this token. all we're doing is you're taking the intermediate neural activations that it's already computed and then running a classifier over these internal computations. And so if you fuse these probes into the inference kernel or the training infrastructure directly, then the overhead is minimal almost no overhead and that's why it's so much cheaper because it can help you screen out the majority of things that you might have otherwise monitored for using a reasoning model and that's what makes it so so effective and so you just don't have to monitor externally in like a very large percentage of scenarios.

[53:15](https://realalpha.app/v/MrnhtyPGCKI/?t=3195) It's easier said than done is the short answer. There's a lot to it. And we're one of the only organizations in the world that really know how to do this. Great. Let's go a bit deeper in some of what you alluded to towards the beginning of this conversation, which is a fundamental question of like, okay, We're able to now go inside the mind, quote, end of quote, of the model and sort of have a glimpse into how they function and what they do. The next question is, so what do we actually do now? Like what actions can we take to minimize all the problems and rewater hacking and all the negative stuff?

[53:59](https://realalpha.app/v/MrnhtyPGCKI/?t=3239) What's the range of things we or labs can do? I think it comes in three, maybe like tiers. The first is monitoring. So how can you just detect bad behavior, you know, before it even happens such that you can just stop the generation or, you know, another common intervention there is you can insert a text prompt in. So like, let's say the model is about to do some dangerous cyber action. You can then just insert into the prompt of the model, like, hey, be really careful here. Don't take a dangerous cyber action. In real time? In real time, correct. So this is like some type of prompt steering. And then you can then reduce the model's chances of taking a harmful cyber action because you've intervened directly into the model.

[54:54](https://realalpha.app/v/MrnhtyPGCKI/?t=3294) So that's step one. It's like, how can you monitor for this behavior? And you already get quite a long way. Step two is kind of at-scale offline anomaly detection and debugging. So the problem statement here is given a vast amount of logs and traces, how do you actually surface the anomalies from all of these traces, all of the problematic behaviors, and then reverse engineer the model to figure out where mechanistically the model has actually gone wrong. So it's like figuring out what the problems are in the model so that you can fix them. And then the third step is what we call intentional design. We think this is really the holy grail of of ai alignment of maybe all of ai training as well where you can actually control generalization in the model and you only get the good stuff in training and none of the bad stuff. So how can you steer and guide the training process so that you can actually design these models with intention and rather by trial and error? So maybe just one little mini example here where let's say you understand the concept of reward hacking. And so you're training the model and you realize that during this gradient step, the model was about to reinforce its idea of reward hacking. You don't want that to happen. You actually want reward hacking to go

[56:32](https://realalpha.app/v/MrnhtyPGCKI/?t=3392) down. And so how do you turn that like increased reward hacking signal and either like remove that entirely and reject that update or like reduce that behavior in the first place? That I think is really what we mean by how do you steer and guide training? How do you steer and guide back crop how can you actually imbue the values the morals of the model that you want with the according to some type of constitution that you that you specify so that instead of having an absent teacher you actually have a a very moral and values aligned teacher teaching your model

[57:15](https://realalpha.app/v/MrnhtyPGCKI/?t=3435) **Eric Ho:** to to be good so you literally give gradient descent a choice that's right yeah we want to give gradient descent a choice so what's the current state uh of this is intentional design

[57:28](https://realalpha.app/v/MrnhtyPGCKI/?t=3448) uh something that you're working on and that's like the next like uh whatever one two three years of of research or is there something that's working today like where what's the state of the art we have a couple rudimentary techniques that work in this like umbrella of intentional design so there's two things that we've published so far um but i'll also hint that there's a lot more exciting stuff coming just around the corner. We have some very, very good internal results here to help with intentional design. But the two techniques that we've published are, one, reinforcement learning from feature rewards. You can essentially take a probe, and you can help, you can optimize against that probe to remove, we showed that we can help remove, like, hallucinations in Gemma, using this as a reward signal. The setup here really matters, though.

[58:26](https://realalpha.app/v/MrnhtyPGCKI/?t=3506) You can't just naively train against a probe, a probe monitor or a probe concept. Otherwise, that just moves this concept into some other part of the model. So you need a relatively sophisticated technique in order to do this correctly. So that's another paper of ours, reinforcement learning with feature rewards. I thought that was like a really interesting first step, but it was kind of like a more simple and rudimentary setup. Another idea is predictive data debugging, where you intervene from the data side. So the problem statement there is how can you predict what your model will learn from a data set before the model even trains on it? And then And what ends up working best is some type of clustering technique where you can cluster data points according to what they will teach the model.

[59:19](https://realalpha.app/v/MrnhtyPGCKI/?t=3559) And you can then just remove the data that you don't want. So we're able to find pockets of data in these public data sets that were quite surprising. Like one of these pockets of data was physics sycophancy. specifically people love to be told that they are great at physics and discovering i think there was some guy out on twitter like two years ago saying like i'm out here discovering new physics like it's for guys like that and models have figured out that uh that people love that and uh you probably I don't want that in your model, so you can just kind of filter that out. Great. We've been talking about Goodfire as a research lab, but you're not just a research lab.

[1:00:09](https://realalpha.app/v/MrnhtyPGCKI/?t=3609) You're a commercial enterprise, venture backed. So how does the business side of the company work? You launch a product called Silico. What does that do? And who do you sell it to? Well, Silico, in short, is our interpretability agent. It can do things like really quickly train a probe. to monitor your model. And so we use Silico as our way to move really, really quickly and essentially help with activation monitoring, training other types of interpreter models that reverse engineer model computations, and to just do interpretability at scale. Um, our customers are the companies who are training and serving models at very, very large scale, typically. So like, we typically do deep partnerships with a relatively few number of customers, where we go, and we provide both like expertise from an interpretability perspective, as well as our interpretability agents and infrastructure, to help them with something like activation monitoring.

[1:01:26](https://realalpha.app/v/MrnhtyPGCKI/?t=3686) And you seem to have a number of customers in biology as well, ARC Institute, Mayo Clinic, Prima Mente. What's the use case there? The use case there is often quite similar. It can be a debugging use case where the model has learned some type of shortcut or some type of problematic behavior. And we can kind of go in and diagnose and remove what that behavior is. Or what's most exciting in life sciences, at least to me, is this concept of finding novel scientific knowledge within these models directly. So one example of this was the work that we did with Prima Mente, where they had a state-of-the-art Alzheimer's diagnostics model.

[1:02:16](https://realalpha.app/v/MrnhtyPGCKI/?t=3736) They didn't know how it worked. They had no idea. It was just a black box. And I was able to diagnose, you know, new patients directly from a blood draw, whether or not they had Alzheimer's.

[1:02:28](https://realalpha.app/v/MrnhtyPGCKI/?t=3748) **Eric Ho:** So we were able to reverse engineer the computations of their model and find a new Alzheimer's biomarker, which ended up just being a fragmentomic biomarker.

[1:02:41](https://realalpha.app/v/MrnhtyPGCKI/?t=3761) So like fragment length ended up being a strong predictor of Alzheimer's disease. and this was very surprising because they didn't know this going in and so we're able to do this type of unsupervised discovery directly in the weights of these models to find novel novel concepts i think this is going to become increasingly interesting across scientific domains especially in the life sciences but also i'd be so curious to try to figure out like Like, you know, when OpenAI's models are solving like, you know, novel mathematical problems, like what's going on inside their mind? There must be something happening there that is quite novel that we can uncover and reverse engineer and see what's happening.

[1:03:37](https://realalpha.app/v/MrnhtyPGCKI/?t=3817) Yeah, and to this point, and as we get towards the end of this conversation and zoom out, I think you predicted maybe last year that by 2028, we'd decode how neural networks work. Do you think that we are closer to this prediction being true? or further away as the state-of-the-art and the astros of the world come online? You know, I was firing from the hip there, but I think we could do it by 2028 still. Yeah, I think we're on track. But what I mean by fully decode these models is given a behavior, can you find causally what drove that behavior from a mechanistic perspective inside the model so it's like given any arbitrary behavior like how does the model do that nothing so for example like an easy behavior to ask the question of is like how does this model do addition we should be able to fully reverse engineer and solve that prior to 2028 including all the way up to very very complex questions such as how does this model solve a millennium problem and I think we can do it.

[1:04:58](https://realalpha.app/v/MrnhtyPGCKI/?t=3898) But this is also assuming a very large compute budget. What I anticipate will happen is not that we're going to want to fully reverse engineer a model prior to its release. I think that would be computationally unwise and probably just not feasible. but you can inspect for the behaviors that you really want to get guarantees on prior to any model release and i think we'll be able to get there before 2028 as well so you can inspect for reward hacking behavior or deception or sycophancy and really drill into what's actually driving that behavior inside the model and be able to do what we want with that behavior. And I think all that will be possible and accessible by our technology to the world before 2028.

[1:06:00](https://realalpha.app/v/MrnhtyPGCKI/?t=3960) And in the meantime, if you're an engineer or an AI builder listening to this, what should you make of this whole conversation? What can you do tomorrow morning to make sure that the products that you create the AI products or AI models you create offer less of an interpretability issue? Is there anything practical and actionable? I think my message to engineers everywhere is these models, you can study them. You can just look at the activations and start trying to understand them and reverse engineer them. And so there are many tools out there that make this possible today, but also just like the field of interpretability and the research and the science aren't mature yet.

[1:06:51](https://realalpha.app/v/MrnhtyPGCKI/?t=4011) Like they're not in a steady state by any means. And so we just need more of the smartest people in the world, like thinking about these problems and learning about them and discussing them. I really want a much richer scientific community around interpretability. And that's part of the reason why we founded the company is because we just wanted more energy behind this concept that I think is so important. And we could also use a lot of help at Goodfire. We're hiring a lot, and especially for strong engineering profiles, infra, machine learning for deployed engineers. And maybe to close, as I reflect on this whole conversation, you know, my takeaway is that a lot of the sort of traditional, quote unquote, methods of monitoring and observing what models do are largely outpaced.

[1:07:54](https://realalpha.app/v/MrnhtyPGCKI/?t=4074) I mean, so people do great work and come up with new ways all the time, but largely outpaced. So on the one hand, on the other hand, interpretability seems to have a key moment right now. And you guys are, you know, a leader, or maybe the leader in the field. At the same time, it's still early. There's a lot to discover. However, what should be the takeaway here in terms of overall safety in the industry? What happens, I guess, in the next, you know, 12 to 24 months? Are we actually at a risk? So maybe my high-level message, and I tell us all the time to the team, is that I want people to believe.

[1:08:38](https://realalpha.app/v/MrnhtyPGCKI/?t=4118) I want people to believe that we can and must solve interpretability, solve alignment, to get precise about what this actually means. And we just need so many more capable, smart, motivated people to be pointed at these problems that I think are the most important problems in the world to solve. And we can do this. I think that there's a lot of pessimism around the risks of AI, which I understand and feel viscerally. But I'm also very optimistic that we can solve these problems and build to a future where we can trust these models. and I think more people should try and really kind of join this effort to solve interpretability, to solve alignment and a lot of it is just believing that we can do this I know we can do this I know we can solve these problems and it'll be hard but I think we're going to be able to solve these very, very critical problems Well, reassuring and inspiring.

[1:09:57](https://realalpha.app/v/MrnhtyPGCKI/?t=4197) Eric, I learned a lot. This was a fantastic conversation. Thank you so much. We appreciate it. Thank you for having me. Yeah, thanks for the thoughtful questions. This was fun. Hi, it's Matt Turck again. Thanks for listening to this episode of the Matt Podcast. If you enjoyed it, we'd be very grateful if you would consider subscribing if you haven't already or leaving a positive review or comment on whichever platform you're watching this or listening to this episode from. This really helps us build a podcast and get great guests. Thanks and see you on the next episode.