Manage agents with feedback and decision points

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 33:23

   And when executing ideas, you become a manager, either of people or agents. And there are so many people who are now not forced, but they have to manage agents because of their work. What are the tips that you will give them, the management tips that they should be deploying? It's the revenge of the MBAs, right? Like I teach at, you know, I teach at a business school. So it's like very, it's gratifying to be like, ha ha, everyone has to be a manager now. All my business people were looking for technical people and now all the technical people in management skills. But actually, those management skills do apply, right? So what you need to, it turns out, I mean, we've been doing a lot of experiments with that at the Generative AI Lab at Wharton, which is what I lead. And a lot of the experiments basically show, you know, look, techniques that work on people work on AI.

2. 34:06

   And so I think what you need to be thinking about is how do you manage a process? Because one of the dangers of the cost of an AI system is assigning a lot of work, not even the token cost, just the effort cost of monitoring work is quite high. So we built all these systems for doing that, right? So if I'm specifying things for an agent, I will specify things like here are the points I want you to check in with me on about the work.

3. 34:30

   Here's where I want you to give me diverse examples that I can select from so I make a choice, right? I'm deciding when to insert my own agency into the system rather than just sending off agents that do work. I am making sure to specify what good and bad looks like in really kind of clear ways. And I'm watching really early on in the process to see whether or not there's a failure in my ability to communicate to the agent, right? Oftentimes they'll head off in a direction that sort of like, you know, Claude has its own obsessions, right? They'll head off in those directions unless you intervene quite early on and often. So, you know, those management skills about thinking about, am I clearly specifying what I need?

4. 35:04

   Am I, you know, am I establishing a feedback loop, right? Am I dealing with, you know, reducing uncertainty over the course of a project? Those things end up really applying here. Interesting. What is the biggest mistake that you think people are making when they're managing those agents? I think they're trusting the agents too much. You'll get an amazing result. Like everyone loves sort of, I do a lot of single prompt, like this is awesome kind of stuff.

5. 35:27

   But the guidance process is incredibly important. And I think that it can be very disempowering, both because the agent knows things you don't, but also because the interfaces we have to work with AI now are increasingly hostile to all. augmentation, and embracing automation, right? They hide the chain of thought, right? And even the chain of thought summaries no longer appear. It's supposed to feel more like magic, and that's disastrous for being a human in the loop.

6. 35:52

   I don't think we can be in the loop anymore, but managers are not really in the loop. They assign work and they have people check in on them. So we have to now create forced positions for the AI to report back to us. Otherwise, they will try and do magic for you, and you don't want that. Can you just grab a process in your own workflow where you can show us how you create this result and how we create the feedback loop so that people can learn?

7. 36:15

   Sure, like I was exploring a sort of random topic that interests me in the humanities, which is, and I've done this for entrepreneurial pitching as well, which is there is a lot of research that stories have shapes, right? And so there is, and so like there's a rise and fall stories and, you know, and like the Cinderella stories. And this is like a longstanding question. And I actually did some research before with primitive LLMs about whether or not we could take entrepreneurial pitches and find the shape of a good pitch.

8. 36:42

   So it's a topic that's really interested me for a long time. And as agents have gotten better, one of the things I've had them do is assign them to explore this topic, right? And so there's a few things there. One of them is, you know, I was assigned them to kind of generate a bunch of ideas on how to do this. But I didn't have it then execute its favorite idea because that would have been bad. So I had to give it feedback part way through.

9. 37:02

   So give me a bunch of ways that you might approach this, figure out what the shape of story research is real. And then, you know, it's like, now speculate for me, just like I would for a student. Give me some thoughts about how you might approach this. Give it feedback again. Great. Why don't you execute one, two, three, and four, first draft of all of them. Show me the results, and then let's pick which example works better from this.

10. 37:21

    Great. I really like what you're doing with three here, but number two has this really nice feature. Let's talk about how we combine those together, right? Why, you know, can you show me where this looks wrong? Can you show me why you think this went wrong? So it's almost like a Socratic method kind of thing of me guiding the AI and giving myself lots of choice points where the really tedious, annoying work, figure out where 5,000 stories are, build the semantic coding tools.

11. 37:46

    I let the AI do. So I get the interesting work for myself. And if there's one message of sort of the latest book and where I'm most desperate to try and communicate with corporate America, it's like we have an automation world that will be bad for everybody in a lot of ways. But an augmentation world is like the dream of work, right? You do the interesting stuff and the AI does the crap stuff and you get to just direct an organization.

12. 38:07

    And it's really fun.