Real Alpha
Watch mode

The MAD Podcast · 2 min · Oct 8, 2026

Text-to-SQL reached 99.5% at a bank, but at great cost

Andy Pavlo

VP of Database Research, ClickHouse

Carnegie Mellon database professor who has led its Database Research Group since 2013 and co-founded OtterTune.

  • Matt TurckHost · General Partner, FirstMark

    Publishes the annual MAD landscape of data and AI; sold TripleHop to Oracle.

About this segment

Pavlo explains why he distrusts text-to-SQL academic benchmarks. A large bank lifted accuracy from 60% to 99.5% with a semantic layer, but it took years and millions of dollars.

From “Andy Pavlo: What Happens When Billions of AI Agents Hit Your Database”, Oct 8, 2026. Watch the full episode

“they were able to get it up to like 99.5%, which is insane. I think that is, I think that's a viable thing now if you invest the time”
Andy Pavloat 24:55 · The MAD Podcast24:55

Takeaways

  1. Benchmarks like Spider are in the training data, so results mislead
  2. The bank's 60% to 99.5% came from a rich semantic and context layer
  3. Off-the-shelf tools won't match that without the same investment
Watch 2:11 from 23:38

In this episode

Full episode
  1. 10:43Agents could mean 10 to 100x more database queries4:37

  2. 23:38Text-to-SQL reached 99.5% at a bank, but at great cost2:11

  3. 33:51LLMs tune databases 85% of the way in 15 minutes2:02Up next

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

  1. 23:43

    So tech to SQL, it's like natural language to SQL. So this is something people have been trying to do since the 70s, right? And every so often there's, you know, maybe a, uh, there's a, you know, some kind of breakthrough in natural language processing people say, okay, now finally it can happen. Right. It hasn't happened except for LLMs. I think, um, the, I've been skeptical of the academic literature, uh, and the research publications on various frameworks to do Texas equal, because oftentimes the benchmarks that are showing spider and a couple other ones, like these are well-known benchmarks that, you know, the models are trained on.

  2. 24:20

    So like, of course, is going to regurgitate the correct answer. I actually was talking to somebody from a very large bank yesterday who said they now have a text-to-SQL interface, but in the initial implementation of it, they were doing something off the shelf and then they further refined it a little bit further in-house, but it was about 60% accuracy, meaning like the LLM or the agent could produce the right SQL query, they produce the right answer 60% of the time.

  3. 24:47

    And that's roughly what I think I've seen before in some other reporting. But they said through this, providing a more rich semantic layer, a context layer, over multiple years and millions and millions of dollars to make this work, they were able to get it up to like 99.5%, which is insane. I think that is, I think that's a viable thing now if you invest the time to have the right semantic layer and the context information you need.

  4. 25:13

    So the model is, has the ALS, sorry, the agent has enough information to derive the right SQL query from the question you're asking. I think, again, that's a large bank with infinite money that they can pull this off. I think if you just download something now, a random person will find that these existing tools are not going to be at that level of proficiency or inaccuracy. But it doesn't mean we can't get there, right?

  5. 25:37

    So now in terms of MCP, the model context protocol, like, I get why they did it. I think this came out of Anthropic