Real Alpha
Try Pro free

Lightcone · 3 min · Jul 30, 2026

A benchmark skill lets agents iterate on performance code

Jeff Dean · Chief Scientist, Google

Jeff Dean and Sanjay encoded their microbenchmark workflow in a skill that measures changes and iterates.

In this episode

Full analysis
  1. 6:01Google built the TPU after estimating speech's server costPro pick 3 min
  2. 25:30Small teams can win where general models stay weakPro pick 6 min
  3. 42:04Fast learned evaluators can change the pace of science4 min
  4. 19:43A benchmark skill lets agents iterate on performance code3 min
  5. 22:08Agent search can make long-running work more reliable3 min
  6. 31:18Clear specs matter more when agents write the code4 min
  7. 36:30Revisiting reliability assumptions can produce new system designs6 min
  8. 47:44Distillation met a need for cheaper model serving2 min
“self-improving, benchmark measurement, benchmark code changes, measure the performance improvement, and then iterate on that.”
Jeff Deanat 20:58 · Lightcone
20:58

Takeaways

  1. Encode the human benchmark sequence as an explicit model skill.
  2. Measure cache footprints and broader benchmarks, not just the target test.
  3. Performance Hints helped models reason better about code optimization.

About this segment

Jeff Dean and Sanjay encoded their microbenchmark workflow in a skill that measures changes and iterates. It shows how a written performance playbook can turn engineering know-how into repeatable agent work.

Jeff Dean

Jeff Dean is Chief Scientist at Google.

Co-founded AI startup Discovery Loop; previously Google chief scientist and co-creator of MapReduce, Bigtable.

  • Diana Hu, Host · Managing partner, Y Combinator

    CTO of Escher Reality; after Niantic bought it, ran AR for Pokémon GO.

From “Jeff Dean: The 1% Rule for Building in AI”, Jul 30, 2026. Watch the full episode

Answers “Are AI agents ready for real work?” See all 104 answers

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

  1. 19:49

    engineering you personally have done? I don't know, skills you wrote, tools that really made a huge difference in your in your workflow? Yeah I mean I guess Sanjay and I were working a few weeks ago And we often do some amount of performance improvement for very low-level libraries. And we have a microbenchmark library we've written at Google where you can write microbenchmarks of how long different kinds of operations take or how long does it take to populate this data structure or whatever.

  2. 20:19

    And sometimes those data structures are used on millions of processes across Google. So it's actually pretty important to make sure they're high-performance. And so you can write micro benchmarks. But then without an agent-based system, what you usually do is you measure what the current performance is on some benchmarks you care about. You make some modifications to improve the performance you hope. Then you rerun the benchmarks, see where things improved. You run maybe a broader set of benchmarks, measure the cache footprint of things.

  3. 20:52

    And so we wrote a skill that basically taught the model how to do most of those things in various sequences so that it could actually do self-improving, benchmark measurement, benchmark code changes, measure the performance improvement, and then iterate on that. and that seemed to work pretty well for some kinds of problems and it really just is us giving the approach we would use as people to the model in a form that it could use.

  4. 21:23

    Wow that seems very impressive. So you're saying you have this skill that if someone got access to it it could do perform optimizations like Jeff Dean. Seems like the world would love this and is worth infinite amount of money? Do someone have access to this? We actually published a document maybe a few months ago called Performance Hints that Sanjay and I wrote. It's like a 30-page document about various kinds of performance tricks.

  5. 21:49

    And some people have taken that and then given it in summarized form to various models and seen that that model can now get better at reasoning about performance issues in code. So you heard it all here. You could actually optimize your own code, But like Jeff Dean, if you take this paper that you published in Performance Hints, it's all free available. So you should all try it. Very cool.