Real Alpha
Watch mode

a16z Podcast · 3 min · Oct 16, 2026

AI query optimization needs benchmarks, not guesswork

Jad Naous

CEO, Grepr

Founded Grepr to cut observability data costs with AI; previously CPO at Imply and an a16z investor.

  • Erik TorenbergHost · General partner, Andreessen Horowitz

    Helped build Product Hunt; founded On Deck, Village Global and Turpentine.

About this segment

Jad says agents can improve query code when grounded in user workloads and performance benchmarks. They still need production experiments to answer questions like how much memory a system needs.

From “Why AI Can Write Code But Can’t Keep It Running (Yet)”, Oct 16, 2026. Watch the full episode

“it was able to ground that in performance benchmarks”
Jad Naousat 29:03 · a16z Podcast29:03

Takeaways

  1. Give agents real workloads and performance benchmarks to ground optimization.
  2. AI can identify inefficient generated query code and propose fixes.
  3. Memory sizing still depends on production experiments.
Watch 2:47 from 27:10

In this episode

Full episode
  1. 17:52AI cut a complex data-lake project from six months to ten days7:06
  2. 53:20Real-world AI takes longer when the environment is unconstrained5:38

  3. 27:10AI query optimization needs benchmarks, not guesswork2:47

  4. 10:48Experienced engineers get more leverage from coding agents4:20Up next

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

  1. 27:23

    Like, how good of a job does it do at that? And the second one is like, okay, now how do we make it reason about this? So let's ask the first one. Like, if I use your method, the JAD method of writing code, and I built some big distributed system, like. How often does it actually get it from performance correctness? How often is it like actually operationally like pretty sensible? And by the way, I work on this.

  2. 27:45

    distributed Gaussian splat renderer, I would say it's shit. Like, it's horrible. Like, because, like, you know, you've got millions of Gaussian splats that you've got to page in and out and stream, and, like, it just totally depends on, like, the running system. And so, like, none of the models do a good job. So in our world, what I've found is that we've defined basically really well what performance benchmarks look like and how the system should run and what the constraints of actually building within like our stream processor look like to get the right performance.

  3. 28:17

    But every once in a while, it'll do, like my latest project has been all about query optimization and query performance. And so I was like, hey. Classic example of something that's like hard. I was like, we are basically generating code that corresponds to SQL and we want to run it in a pipeline. Yeah. And we're like, okay, well, you're generating shit code, fix it. Yeah. And it started basically going through, and what it didn't realize was that it wasn't actually doing initially any optimization on the code.

  4. 28:47

    It's like, oh, it's fine, it looks like, what's the problem? And I said, well, look, there's these conjuncts over here, you're doing and, and this particular term is repeated, and it should have been taken away. And then it's like, ah, okay, let me figure this out, and so on. And then, like, it basically went on a thread to try to figure out what are all, like, it built a bunch of hypotheses around, like, what are all the things that it can go and optimize to make this better.

  5. 29:14

    But ultimately, it was able, because it was able to ground that in performance benchmarks, and I was like, look, here's a data set of the queries and, like, what the users are asking and so on. Go work on it. And so it can basically do a relatively good job that way. But when it comes to saying something around like, how much memory should I give the system? Yeah. It has no idea. And that has to come from actual experiments that run in production.

  6. 29:46

    Yeah.