Agents could replace bespoke apps with exposed APIs

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 9:15

   whatsapp portal tool of the automation yeah is there something that you have done personally with your claws beyond software engineering that you think is fun or interesting? Yeah, so in January, I had a claw. I went through a period of claw psychosis. So I built, I have a claw basically that takes care of my home and I call him Dobby the Elf Claw. And basically I used the agents to find all of the smart home subsystems of my home on the local area network, which I was kind of surprised that worked out of the box.

2. 9:46

   Like I just told it that I think I have Sonos at home. Like, can you try to find it? And it goes and like IP scan of all the, basically, computers on the local area network, and it found the Sonos thing, the Sonos system, and it turned out that there's no password. And it's like, oh yeah, you have these Sonos systems installed. Let me try to reverse engineer how it's working. It does some web searches and it finds like, okay, these are the API endpoints. And then it's like, do you want to try it? And I'm like, whoa, like you just did that. And I'm like, yeah, can you try to play something in the study? And it does. And music comes out. And I'm like, I can't believe I just...

3. 10:18

   That's crazy. That's like three prompts. I can't believe I just typed in, like, can you find my Sonos? And that suddenly it's playing music. And it did the same for lights. And so basically, like, it kind of hacked in, figured out the whole thing, created APIs, created a dashboard so I could see the command kind of center of like all of my lights in the home. And then it was like switching lights on and off.

4. 10:35

   And, you know, so I can ask it, like, don't be at sleepy time. And when it's sleepy time, that just means all the lights go off, et cetera, and so on. So it controls all of my lights, my HVAC, my shades, the pool and the spa, and also my security system. So I have a camera pointed outside of the house. And anytime someone rolls in, I have a Quinn model that looks at the videos. So first of all, there's change detection.

5. 10:58

   And then based on change detection, it goes to Quinn. And then it actually tells me, it sends me a text to my WhatsApp. It shows an image from the outside. And it says, hey, a FedEx truck just pulled up. FedEx truck just pulled up. And you might want to check it. And you got a new mail or something like that. And Dobby just texts me this. It's really incredible. um so so dobi is in charge of the house i text through with it through whatsapp um and it's been like really fun to have these macro actions that maintain my house i haven't like really pushed it uh like way more beyond that and i think people are doing a lot more crazy things with it but for me even just a home automation setup i used to use like six apps yeah completely different apps and i don't have to use these apps anymore like dobi controls everything in natural language

6. 11:37

   it's amazing um and so i think like i haven't even pushed a paradigm fully but already that is so helpful and so inspiring, I would say. Do you think that's indicative of like what people want from a user experience perspective with software, right? Because I don't think, you know, it's pretty ignored that it takes humans' effort to like learn new software, like new UI. Yeah, I think to some extent, that's right.

7. 12:00

   It's like working backwards from how people think an AI should be because what people have in their mind of like what an AI is, is not actually what an LLM is by like in a raw sense. Like LLM is a token generator, you know, like more tokens come out. But what they think of is like this persona identity that they can tell stuff and it remembers it, you know, and it's just kind of an entity behind the WhatsApp.

8. 12:20

   It's like a lot more understandable. So I think to some extent, it's like matching the expectations that humans already have for what DNA should behave. But under the hood, there's like a lot of technical details go into that. And LLMs are too raw of a primitive to actually type check as AI, I think, for most people, if that makes sense. Yeah, I think that's like how we understand what the AI is and like the description of it as DABI or some personality obviously resonates with people.

9. 12:47

   I also think that the unification that you did across your six different software systems for your home automation speaks to a different question of like, do people really want all the software that we have today? Yeah. Right. Because I would argue like, well, you have the hardware, but you've now thrown away the software or the UX layer of it. Do you think that's what people want? Yeah, I think there's this like, there's this sense that these apps that are in the app store for using these smart home devices, etc.

10. 13:16

    These shouldn't even exist kind of in a certain sense. Like, shouldn't it just be APIs and shouldn't agents be just using it directly? And I can do all kinds of home automation stuff that any individual app will not be able to do, right? And an LLM can actually drive the tools and call all the right tools and do pretty complicated things. And so in a certain sense, it does point to this, like maybe there's like an overproduction of lots of custom bespoke apps that shouldn't exist because agents kind of like crumble them up and everything should be a lot more just like exposed API endpoints.

11. 13:48

    And agents are the glue of the intelligence that actually like tool calls all the parts. Another example is like my treadmill. There's an app for my treadmill and I wanted to like keep track of how often I do my cardio. But like, I don't want to like log into a web UI and go through a flow and et cetera. Like all this should just be like make APIs available. And this is kind of, you know, going towards the agentic sort of web or like agent first tools and all this kind of stuff.

12. 14:13

    So I think the industry just has to reconfigure in so many ways that it's like the customer is not the human anymore. It's like agents who are acting on behalf of humans. and this refactoring will probably be substantial in a certain sense. One way that people sometimes push back on this is like, do we expect people to Vibe code some of these tools? Do we expect normal people to do this kind of stuff that I described?

13. 14:33

    But I think to some extent, this is just, you know, technology as it exists today. And right now there is some Vibe coding and I'm actually watching it and I'm working with the system. But I kind of feel like this kind of stuff that I just talked about, this should be free, like in a year or two or three. There's no Vibe coding involved. This is trivial. This is table stakes. This is like any AI, even the open source models, et cetera, can like do this.

14. 14:53

    You should be able to translate from a less technical human's intent very easily to this outcome. Yeah, extremely easily. yeah today it's vibe coding and it's involved and not many people are going to do it but and you still have to make some design decisions right we were talking about like will you take frames for example yeah yeah but i kind of feel like this will just uh start to the barrier will just come down and it's just ephemeral software on your behalf and some kind of like claw is handling all the details for you but you're not involved claw has a claw has a machine and it will figure it out and it's just presenting you uis and you're like saying stuff you know why haven't you um i I guess, like push the boundaries of what you can do personally with CLAWS?

15. 15:32

    Like, is it, you know,