Watch modeSubscribed

Knuckle Up with Nakul · 3 min · Aug 25, 2026

Security review costs can vary sevenfold across models

Vercel benchmarks models for security work as capabilities and prices shift. Kimi was less capable than Sol but seven times cheaper, making model choice an operating-budget decision.

Guillermo Rauch

Guillermo Rauch is founder and CEO of Vercel.

Founded Vercel, which created Next.js and the v0 AI tool for building websites.

Nakul Mandan, Host · Host, Knuckle Up with Nakul
General partner at Audacious Ventures; his show goes deep with founder CEOs on how they build.

About this segment

Vercel benchmarks models for security work as capabilities and prices shift. Kimi was less capable than Sol but seven times cheaper, making model choice an operating-budget decision.

From “Inside Vercel's AI-era playbook: "Recursive Founder Mode" | Guillermo Rauch (CEO)”, Aug 25, 2026. Watch the full episode

Takeaways

DeepSec prompts models to look for security vulnerabilities.
Benchmark results change as new models are released.
Model choice affects security coverage and token spend.

In this episode

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

48:36
because it literally has become almost every other day. And so we released a deep sec bench because the talk of town right now is cybersecurity. And the question that I get a lot from CIOs and CISOs is, how do I protect myself? how worried should I be about the threats from mythos or open weight models. And I myself was finding myself asking our CTO who created this tool DeepSec, which allows you to run.
49:06
Basically, it's really interesting. So agents, much like human beings, can take on roles. You know, when you go home and you're with your kids, I'm not the intense CEO, like hopefully for them, right? Like I'm like playful and whatever. And when I went to the World Cup final, I was a soccer fan rooting for Argentina. And now I'm the CEO. And the same happens with models. So DeepSec tells every agent and every model, go and be extremely paranoid and pay more attention to auth and security and privacy and try to find vulnerabilities.
49:45
And you would be shocked at how good the models can be at analyzing themselves. It was a little bit like psychotherapy because they wrote the code and then they can tell you how the code is wrong. And what was happening is every model release was changing the economic equation. We noticed, for example, that Sol is incredible at finding vulnerabilities, but it was costing for our code bases thousands and thousands and thousands of dollars.
50:13
And then Kimi came out and we realized, oh, it's not quite as good as Sol, but it's seven times cheaper. And we need to adjust this priors every day because there's a new open weight release. There's been three or four open weight American models in just the last couple of weeks. And so we've sort of transferred our, you know, instead of me having to vibe out what is the right agent or model, we're trying to formalize that.
50:40
And then we'll try to, hopefully soon when you ask V, hey, can you do a security review of this? It's automatically calling DeepSec. And DeepSec itself is going to say, all right, how much do I invest? Because the question about intelligence on tap is that, yes, it's like tap water, but imagine that you have several faucets and it's also liquid gold. And are you going to have just like a casual liquid gold sprawl for this task?
51:11
And so much of this, and this is why, you know, for people that are thinking about the future of their companies, so much of this is what's going to inform your P&L. Tokens are going to become a huge percentage of your operating budget. And so you're going to have to think about, well, for this area of the business, we spend this much with these models. For this part of the business, we spend this much.
51:37
It's going to vary tremendously.