Semantic checks could make agent software safer

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 22:03

   What are like some that come to mind that like of these future areas? What I realized is that a lot of people are still stuck in like fairly basic patterns and we have like a gigantic job to both push the frontiers of model intelligence, reliability, all of these things that we really care about. as well as education to push the limits of software. And there's a thing in the intersection of education and pushing the frontier forward, which is this collaboration between research and product, which is, I think, where all gigantic ML wins come from.

2. 22:37

   And this, I think, is kind of an unexplored place. If we had more help, we could do so much more here to encourage people in the world or even fund the kinds of things that we think should exist and really allow people to push the frontiers of what software can possibly do. So I'll give a couple of examples of this. And, you know, this is going to be a little bit of a spoiler for a launch, a release we will have sometime in the future.

3. 23:02

   It is that we want to, like, fund these things. And there's so many cool shapes of stuff missing. Like, one of the things people know I'm passionate about is coding agents that are free from the tyranny of the KVCache. I think that so much coding agent is built around the KV cache. And the KV cache is freaking awesome, too. I don't want to toss it away, but I want to be aware of it and design around it and make different sets of trade-offs to make things that were pure sci-fi possible.

4. 23:30

   Another thing that I think would be so unbelievably cool is... Do you know what dependent typing is? Dependent typing is a lot about adding constraints to your programs that are like higher level than purely the type system. You know, like you can make sure that your function returns an int, but dependent typing would allow you to do things like it is an int greater than zero. And then now downstream applications can use those properties to have their own sets of guarantees, et cetera.

5. 24:00

   To get higher and higher levels of composability with your software, because they're more legible, and they don't require as much unit testing edge cases, and also as much human understanding to understand the whole system. And one of the things I think would be so freaking cool would be a dependent, a semantic dependent typing, where instead of having to represent your outputs as pure types, like int, float, you could actually say specific properties of that in natural language, because there's so many properties you could care about in software, and have a type system that uses JEV-like models verify that everything works together.

6. 24:33

   I think that that would be so unbelievably cool. Maybe it doesn't matter in a world where everyone uses coding agents, but I think that we should do stuff like that just because it's awesome. And that's what early internet energy is about to me. So security is a huge thing as well. As these models get cheaper, faster, all over the place, where can we add extra levels of security kind of to avoid if you guys saw the news about like the cohere quen 3.8 next flash debatably makes a sketchy call to like phone home it's debatable i'm not going to talk about whether or not that's legit or not because i haven't read enough about it but why can't we just have checks on top of every single tool call that actually are semantic to your organization rather than whatever anthropic thinks should go into their auto router you know like maybe in this part of the code base they shouldn't be making any

7. 25:24

   calls to this other part of the code base at all and how do you add these checks uh you know you need like extremely fast and cheap security all over the place which could be super awesome you know i'm not saying everyone should build it right now but i think that we should collaborate with you know the making pushing capabilities and software in order for like thinking it's so interesting because a lot of those problems like the problem you're talking about now like you could imagine you know approaching from the perspective of a model company or like there's all these you know enterprise agent security companies that are popping up right that aren't model builders themselves but they're like yeah let's put in some like you know uh some actual like deterministic software to to monitor these things and it's interesting to like think about

8. 26:03

   you approaching a similar set of problems but more from like a model perspective uh versus obviously there's plenty of other folks that are just trying to build software that you can put around in lll so my philosophy is that we should enable those software players to win and just software is so fucking cool exactly but ideally they also tell us that we don't want their data but they should tell us like hey you have jaggedness on this particular axis that makes our thing less reliable let's work together and actually they could just tell us like we are not going to tell you what we work on at all here's the rough shape of this task fix it and i would just be like hell yeah thank you so much for telling us this hopefully it solves your problem and a whole bunch of other people's problems i hope you produce tons of value in

9. 26:45

   the world like i hope you make like a decent amount of money and i hope that this enables other people to build all sorts of other things because now this jaggedness thing is healed and