Agent training data makes old APIs harder to retire

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 52:55

   Because, like, early on, I mean, even now, like, you guys focus a lot on that. Yeah. And you've talked a lot about that. Does developer experience even matter anymore? I think it still matters. And I think you're sort of, you know, like the context, I think you're official, like the thing you're talking about is like now with agents, right? Like does developer experience. Like the docs are already there.

2. 53:14

   The agents are going to look at the docs, like all that stuff. To that point though, like almost all the things that make developer experience good also makes the agent experience good. It's kind of what we've seen in practice, which is like things like having good docs, having descriptive error messages, having fast feedback loops, right? Like, I do think there's certain things have changed. The other day I was like setting something up with Cloudflare and it's like, I find Cloudflare like notoriously annoying to work with sometimes.

3. 53:42

   And I threw an agent at it and it was able to like just configure a bunch of like, ask, you know, things in Pulumi for me. And, you know, and so I will say like agents does, they do make it a little bit easier to work with things that are like harder to work with in the past, you know, So like the sort of human toll of dealing with YAML has arguably gone down a little bit because you can just like, you know, tell your agent to deal with it.

4. 54:05

   But for the most part, I think, you know, the things that make developer experience good also make agent experience good. I think to some extent also like infrastructure over time anyway becomes and needs to be more programmatic. I think that's always been a thesis about sandboxes is that, you know, like things that you previously had to configure using YAML, that should just be an SDK. that should be more like imperative like launch a sandbox like do stuff with it tear the sandbox down right like rather than sort of declarative using using you know some sort of dsl it also pushes like a little bit of a different interface that maybe previously didn't exist i think in either way like sdks are as relevant as they as they always were if not more is there anything you changed in your product just because like you know most people are just running agents on

5. 54:52

   on it? We do. I mean, we also, we actually do run a lot of internal tests where we have agents, we tell agents to like build a modal app and run it. And I mean, a very concrete thing that is kind of funny is that we've actually seen that like because of agent training data, it's very hard to deprecate things now. We see that like agents often like love using, you know, old versions of modal syntax and it's kind of hard to nudge them away from that.

6. 55:20

   So we've actually had to be a little bit more conservative going forward. Like, you know, over the last year or so with how we deprecate and remove features, it takes, we have to be very careful to update every single example that's out there online. Yeah. And, you know. You have skills, right? Yeah, there's like stuff like that. But like, there's also an insane amount of like modal code that's now like online too, that goes into the training data.

7. 55:42

   I've seen like that to be like kind of a, you know, a little bit of like an interesting example where like, another thing, by the way, too, is like, there's obviously like to sort of similar to like search engine optimization there's like agent engine you know whatever people call it agent optimization we often like run internal benchmarks so we're like ask an agent, like, hey, like, can you deploy a stable diffusion for us?

8. 56:05

   Use whatever tool you want. And we see if it recommends modal or not. And those things are very useful for us to sort of track over time. Or like, hey, spin up some sandboxes. Or like, hey, like, rank all the sandbox vendors. So we spend a lot of time, like, doing those things to make sure. I mean, what do you do about that? Yeah, you say, hey, like, you know, build a stable diffusion thing. Does it recommend modal or not?

9. 56:24

   Like, yeah, what do you do about that? Yeah, I think that's harder. I think, you know, it's like search engine optimization. I think it's primarily through content. It's like you just have to put out a lot of content. You can also, to some extent, like sort of see the perception, I think a little bit more. Like you can ask an agent, like, you know, how does, you know, rank all the sandbox vendors by reliability or rank all the sandbox vendors by price and sort of see.

10. 56:51

    And it does feel like agents are a little bit more prone to like misunderstanding pricing and, you know, things like that. So we also spend a lot of time like just making content more clear and more like, you know, understandable for agents because we know they're all indexing like everything that's out there and that's talking about modal.