Open-ended search can find ideas direct optimization misses

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 5:45

   So on open-endedness, tell us a little bit more about your research direction with open-endedness. Like, do you think that other labs aren't focusing on this as much? We think so. you know open-endedness is kind of this idea that sometimes if you just go directly after the goal you're not thinking enough about clever ideas and you know there are different examples of this but you know imagine you started cooking food over fire and you're like let's make it much faster to heat up food if you just try to like deal with the fire maybe you create some like pressure with the fire, but you would never invent a microwave. The microwave happened because other people started working on radio waves and then realized, oh, wow, my like this, this chocolate bar in my pocket is melting. What's up with that while working on radio waves and telescopes and things

2. 6:40

   like that? And then, you know, you say, oh, maybe there's something here and then you combine it and then you invent the microwave. And so this idea that you combine interestingly different ideas from different branches, similar to evolution and biology, that is something that we think is very important. And then there are a lot of interesting subtleties on how you make that work. One of the earliest papers that has thought about this is the Darwin-Gödel machine also. Four of the five authors of that paper are at recursive as well. And in the Darwin-Gödel machine, they also asked agents to implement their next versions of those agents. And you see the kinds of plots that then became much more popular with Andrej Karpathy's auto research project. But you see those plots first in that paper of the Darwin Gödel machine. And so we've been at it as a founding team,

3. 7:34

   in some cases for decades. Jeff Kloon, one of our co-founders, has been thinking about open-endedness. Tim Rock-Teschel has done it. I'll give you another interesting example. Imagine you're trying to make an LM safer in terms of, you know, different, like, prompt injection attacks and whatnot to get it to say stuff you don't want it to say. Well, you can just have a human define, like, here's my benchmark data set, you know, I had these ideas, and then you try to, you know, test Whether it fails on the eye, the eye gets smarter, and you get 100 out of 100 correctly not attackable examples or something like that.

4. 8:09

   But an open-ended approach would actually be you ask another LLM to attack the first one. And once it figured out how to do the attack, you then inoculate it, and then you ask it again. And so now you basically can co-evolve an attacker LLM with a defender LLM. And that idea is called rainbow teaming, as invented by Tim Rock-Teschel and his team at Google DeepMind. And it's one of those many very powerful ideas where if you create the right environment and you allow the AI to co-evolve, given the rewards that you give it, more interesting things happen that you may not have directly anticipated.

5. 8:52

   it. And we believe that's a better approach long term to model this community of AI scientists