Smallville tested whether agents could form unscripted behavior

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 3:59

   scale really is going to look like and so this all started with Smallville it did okay explain what that is and like your whole background behind that for sure so Smallville was an experiment that we run back in 2023 and 2022 where we had the initial observation that these large language models initially we were using GPT-3 and then GPT-3.5 the models were still fairly janky but you could see a glimpse of opportunity where these models were trained on such broad human data that when you poke at the right angle, you can actually extract a lot of human data that appeared in its training data. This might be from the web, Wikipedia, some of the social media.

2. 4:43

   And what we really start to ask ourselves is, can we actually use this to create an interaction paradigm that is truly new, that goes beyond just simply generating context or classifying different tasks, but can we actually generate truly new behaviors? And we decided that the most ambitious version of this is actually to create a simulation of an entire person. And then once we had an entire person simulated, can we actually go create an entire community?

3. 5:11

   So Smallville is an experiment that came out of that effort. So it was a game town. Computer scientists love games because it's an amazing way to demonstrate new capabilities. So we created a game town that looks a lot like what pokemon sims like these kind of towns might look like and then we populated this game town with 25 agents and these agents all had different names different characteristics different life stories and they were basically language models paired with what we called back then an agentic architecture that had memory planning and reflection so they would actually wake up in the morning do their routines go to work have relationships the simulation itself actually sets in place, takes place the day before Valentine's Day.

4. 5:53

   So these agents will actually come together to have party. So there is a character that's actually organizing Valentine's Day party. There are characters that are asking themselves out. And none of this was pre-programmed, but what we could see is that we can basically create simulations of these people with emerging phenomena just coming out of the simulation. So that was the experiment that we ran in 2023.

5. 6:16

   That was Smallville. That particular work ended up introducing a few key concepts. One was that we can create simulations of people. And the other was the core agentic architecture and that we can create agents using generative AI models. It was very early days of these agentic workflows so we could really start to see them becoming possible. And then we've been building on the technology basically for us to then create simulations that are more and more accurate and that's more diverse and that's bigger in scale.

6. 6:45

   How do you test the validity of this?