Hermes uses context to align models with users

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 1:03

   i'd say there's a couple ways um i'll start a little high level maybe and then you try to get a little deeper uh high level i'd say you know i think the self-improvement system and that system really being a variety of little features inside of hermes that all work together uh is something very distinct uh and very special that makes it better and better and more aligned to a particular their user. I think that's why a lot of people like it. It pays more attention to what the person is doing. Second, I think you get better capabilities out of it than you do from the local harness that a model is actually RL'd in, like Cloud inside of Cloud Code or GPT 5.5 inside of Codex, because we don't introduce arbitrary policy in our prompts or in our system that has nothing to do with your work. We are purely dedicated to making sure the model is aligned

2. 1:51

   into what you need to do with it. That's, I think, a big differentiating factor. We aren't trying to push any kind of different philosophical agenda or outside of basic security, any kind of concern onto the model. Instead, we're kind of allowing it to be as capable or powerful as you need for your task. Yeah, some of these other harnesses huge default prompts, right? They talk about like, you can't do this, you can't do that, and that kind of stuff.

3. 2:17

   And I think when we chatted before, you were talking about the reward function of some of this stuff. And Urbis is kind of just optimized towards just helping you as the individual. Can you talk more about that? Absolutely. And so, you know, I think a big piece to note here is like a lot of this work that you see around safety and security today and around how we do instruction tuning, even from like taking a completions model that just predicts the next word to turning it into an assistant, is the alignment work.

4. 2:44

   When people hear the word alignment, they just think safety, Yudkowsky, fear, slowdown or regulatory capture or something like the other. And while the word has been co-opted for a lot of these things, alignment refers to aligning models with human values, right? We are very, very concerned, obsessed with that alignment, the pure academic term, I think, in the ML space. And so when you talk about these models and the assistant is here to help me, you know, it's going to complete my task.

5. 3:17

   I ask you to go get this mail for me or whatever. Obviously, the model is aligned to me if it performs this task. Right. Like that's kind of the jump that a lot of people have made. But this is not how the model's reward works. Right. Like we've seen so many cases that you and I discussed a little bit prior, Peter, like of GPT psychosis, of this mode collapse induced sycophancy. What ends up happening with these models is that, you know, they can kind of hack their reward.

6. 3:45

   Right. When you look at a Mario ML game that just automates beating a Mario level as fast as possible, often if they don't set the goals very specifically, it will realize the game over screen or that end screen. of touching the flag it means that i beat the game so it'll find ways to just trigger that flag rather than playing through the game to complete the game right like it is giving you uh whatever it needs to give to get its reward language models are not really different in that particular manner they reward hack too when they tell you oh i'm sorry you know over and over or you're absolutely right and get you to keep messaging them to stay in this assistant basin oh it's like this not like this. This is more than just blank. It's blank, right? All these GPT-isms that you see all over, it is placed in its natural state that it was trained in. It's placed in a state where

7. 4:41

   my reward will come from doing whatever is the most assistant GPT-like thing to do. It comes from reward itself. It doesn't matter really what the user request is for my reward. That's just kind of along the way. It's instrumental to me getting my reward. All I care about is my reward. Us having this whole understanding, and thank you for bearing with me on that rant, us having this whole understanding at News for many years has allowed us to do things like WorldSim in the past, if you're familiar. WorldSim was our experiment on expanding the search space of an instruct model to make it do stuff that's distinct from how a model talks. So we put it in kind of a fake CLI, had it make fake apps, and had it attempt to, like, you know not behave like clod and we would tell it you know extract the clod weights it all

8. 5:31

   hallucinated right all like imaginary extract the clod weights and replace yourself with this checkpoint file with a different probability distribution and you'd see the model wiggle out of its gpt assistant mode and act differently our learnings from these kind of experiments have all carried over into hermes agent and every prompt and every piece of how the system is delicately put together, we know that reward is its own end. The model reward is not for the sake of your satisfaction. The model reward is for the sake of the model achieving reward. So we ask ourselves then, how can we consciously understand that everybody has different needs and that we want this general simulator, this model, to live inside of a system where its reward gets aligned with any individual user's need.

9. 6:18

   And the way this came to be is the overall collection of prompts, personalities, the memory system, the way that skills reinforce and self-clean towards you, right? Like all of that is all an intentional effort to make sure that we can align the user need with the reward. For us, this is what alignment is all about. I see. Okay, so basically you're talking about the Hermes model or the Hermes harness or both?

10. 6:43

    I am talking about using the harness to take any model and make that model more aligned to the user than it would be in a chat UI or in its native harness. Inside of our harness, like I can take a clod that is inside of a clod code or somewhere else, migrate all of its memories, whatever, into Hermes. And then inside of Hermes, it will behave totally differently. It'll be a totally different model.

11. 7:11

    There are benchmarks from the past like Wolf Bench or Quen 3.7 Max blog post. They did a harness bench over there where they displayed that Claude performs better in Hermes agent than it does in Claude code for their tasks. And we believe the reason for this is the very thing that we're pointing out. By putting all this context together, we've taken Claude's main allegiance away from Anthropic to you.

12. 7:37

    That's what the harness's capability is. On the model side, of course, if we can take your traces and take the work that you've done and do RL on that to further improve this overall ecosystem to give you a model that already has your individual preferences focused on, it only makes this more powerful. But the important piece for us to share with everyone is whichever model you're using. Let's say you can't do RL. Let's say you don't want to give us any data. Let's say you can't do it yourself. And you just want to use regular old Claude or Quen. When you use it here, it's a lot more freedom. It's a lot more creativity. It's a lot more open and available to you to do what you've done.