Real Alpha
Try Pro free

Lightcone · 3 min · Jul 30, 2026

Agent search can make long-running work more reliable

Jeff Dean · Chief Scientist, Google

Jeff Dean says agents falter when tasks move beyond what they know, and describes using skills and multiple candidate approaches with evaluators.

In this episode

Full analysis
  1. 6:01Google built the TPU after estimating speech's server costPro pick 3 min
  2. 25:30Small teams can win where general models stay weakPro pick 6 min
  3. 42:04Fast learned evaluators can change the pace of science4 min
  4. 19:43A benchmark skill lets agents iterate on performance code3 min
  5. 22:08Agent search can make long-running work more reliable3 min
  6. 31:18Clear specs matter more when agents write the code4 min
  7. 36:30Revisiting reliability assumptions can produce new system designs6 min
  8. 47:44Distillation met a need for cheaper model serving2 min
“stick to the ones that seem most promising and discard the ones that didn't seem to work”
Jeff Deanat 23:58 · Lightcone
23:58

Takeaways

  1. Agent performance degrades when tasks leave its familiar territory.
  2. Skills and hints can keep work on better-understood paths.
  3. Use multiple approaches and evaluators to find promising solutions.

About this segment

Jeff Dean says agents falter when tasks move beyond what they know, and describes using skills and multiple candidate approaches with evaluators. Searching and discarding weak paths can improve reliability.

Jeff Dean

Jeff Dean is Chief Scientist at Google.

Co-founded AI startup Discovery Loop; previously Google chief scientist and co-creator of MapReduce, Bigtable.

  • Diana Hu, Host · Managing partner, Y Combinator

    CTO of Escher Reality; after Niantic bought it, ran AR for Pokémon GO.

From “Jeff Dean: The 1% Rule for Building in AI”, Jul 30, 2026. Watch the full episode

Answers “Are AI agents ready for real work?” See all 104 answers

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

  1. 22:13

    Now we talked about agents. Everyone here is probably building one or built one at some point. And I'm sure everyone has seen your agent go off the rail at perhaps like step 30 or 40. Like agents are great for like up to step, I don't know, 10 or something and then gets shaky at step 50. What do you think is the constraint today? Is it like context, evaluators, or just errors that compound because it's basically an open system?

  2. 22:41

    Yeah, I mean, obviously we want agents to be able to run for very long periods of time because that's how they're going to solve more and more complicated problems. But as you observe, today, they sometimes stop working after 10 interactions with the tools and so on. and sometimes that's because the model is trying to do something it doesn't have a lot of experience doing so it's been trained on a whole set of things and as soon as you get a little bit off the distribution of things it knows how to do then like most machine learning models it will you know its performance will suddenly do will start to degrade and the farther you get off the comfort zone of what it knows how to do the the more likely it is to to not work as well so there's a bunch of things you can do so one is you know give the model skills and hints that

  3. 23:34

    kind of tend to keep it in in the sort of more brightly lit path of things it does know how to do I think you know having multi-agent systems where you have multiple agents trying different approaches and you can evaluate you have maybe another model or another agent that's evaluating which ones of those seem promising is another way to kind of, in some sense, search the path, search the space of possible solutions and stick to the ones that seem most promising and discard the ones that didn't seem to work or maybe that went off the rails or whatever.

  4. 24:12

    And that's a very, very useful general technique is inference time compute to perform search over plausible ways of solving the problem that can get much, much higher performance or much more reliability in long-running agent flows. How are some ways you implemented this particular workflow for your agents internally? Yeah, I mean, we have, you know, harnesses and then we have a whole set of skills, particularly in the internal Google development environment.

  5. 24:43

    We have skills so that the agents can know how to use lots of our internal tooling for coding or for code reviews or for, you know, measuring performance or, you know, fetching log files. and those are just skills that you can add to make the base model more capable even though it hasn't necessarily been trained on exactly the way that you know google internal engineers would fetch log files from our you know proprietary system with the right kind of skill definition you can actually get it to work and that that improves the usefulness of the agents now let's Let's talk about where startups can win.