Good developer experience also helps coding agents

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 53:11

   Because early on, I mean, even now, you guys focus a lot on that. You've talked a lot about that. Does developer experience even matter anymore? I think it still matters. And I think you're sort of, you know, like the context, I think the thing you're talking about is like now with agents, right? Yeah. Like the docs are already there. The agents are going to look at the docs, like all that stuff. To that point, though, like almost all the things that make developer experience good also makes the agent experience good.

2. 53:40

   It's kind of what we've seen in practice, which is like things like having good docs, having descriptive error messages, having fast feedback loops. Right. Like I do think there's certain things have changed. The other day I was like setting something up with Cloudflare and it's like I found Cloudflare like notoriously annoying to work with sometimes. And I threw an agent at it, and it was able to just configure a bunch of things in Pulumi for me.

3. 54:05

   And so I will say agents, they do make it a little bit easier to work with things that are harder to work with in the past. Like the sort of human toll of dealing with YAML has arguably gone down a little bit, because you can just tell your agent to deal with it. But for the most part, I think the things that make developer experience good also make agent experience good. I think to some extent also like infrastructure over time anyway becomes and needs to be more programmatic.

4. 54:35

   I think that's always been a thesis about sandboxes is that, you know, like things that you previously had to configure using YAML, that should just be an SDK. That should be more like imperative, like launch a sandbox, like do stuff with it, tear the sandbox down, right? Like rather than sort of declarative using, you know, some sort of DSL. It also pushes like a little bit of a different interface that maybe previously didn't exist.

5. 54:56

   I think in either way, like SDKs are as relevant as they always were, if not more. Is there anything you changed in your product just because like, you know, most people are just running agents on it? We do. I mean, we also, we actually do run a lot of internal tests where we have agents, we tell agents to like build a mobile app and run it. And I mean, a very concrete thing that is kind of funny is that we've actually seen that because of agent training data, it's very hard to deprecate things now.

6. 55:27

   We see that agents often love using old versions of modal syntax, and it's kind of hard to nudge them away from that. So we've actually had to be a little bit more conservative going forward over the last year or so with how we deprecate and remove features. We have to be very careful to update every single example that's out there online. You have skills, right? Yeah, there's stuff like that. But there's also an insane amount of modal code that's now online too that goes into the training data.

7. 55:58

   I've seen that to be a little bit of an interesting example. Another thing, by the way, too, is obviously similar to search engine optimization. There's agent engine, whatever people call it, agent optimization. we often like run internal benchmarks we're like ask an agent like hey like can you deploy a stable diffusion for us use whatever tool you want and we see if it recommends modal or not and those things are very useful for us to sort of track over time or like hey spin up some sandboxes or like hey like rank all the sandbox vendors so we spend a lot of time like doing those things to make sure what do you do about that yeah you say hey like you know build a stable diffusion thing does it recommend modal or not like Yeah, what do you do about that?

8. 56:41

   Yeah, I think that's harder. I think, you know, it's like search engine optimization. I think it's primarily through content. It's like you just have to put out a lot of content. You can also, to some extent, like sort of see the perception, I think, a little bit more. Like you can ask an agent, like, you know, how does, you know, rank all the sandbox vendors by reliability or rank all the sandbox vendors by price and sort of see.

9. 57:06

   and it does feel like agents are a little bit more prone to like mis-understanding pricing and, you know, things like that. So we also spend a lot of time like just making content more clear and more like, you know, understandable for agents because we know they're all indexing, like everything that's out there and that's talking about modal. Have you thought about just like making a bunch of GitHub repos and having the agents like write a bunch of modal code in there?

10. 57:32

    Have you tried that? Slop cannon.