One real task convinced Wu that Devin’s approach could work

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 30:18

   When you started Cognition, did you know that you were going to try to make an automated software engineer? Was that your first idea? I'll say it was always two things. One, it was always related to code and software, which again probably has something to do with us all being programming nerds. And then two is it was always around the idea that these would be like real multi-step iterative processes which was a, I would say was a real hot take at the time.

2. 30:45

   Like this was 2023. Okay, so this is when I met you guys and I was on a walk with a mutual friend of ours and I remember first here, this is, you guys scaled from what like like, a million in revenue to, like, 500 million or something like that in, what, like, 20 months or something? Yeah, yeah, a little more than 18 months. I remember before, I got no revenue, and I heard the idea, and it was pitched as, like, essentially, like, an automated software engineer.

3. 31:05

   And I was like, holy shit. It was, like, a huge—you're going after labor. This is a huge market. And then you released— Yeah. I remember the demo video. Yeah. And then you got a lot of shit. Yeah, yeah, yeah. Is that a quick thread? Well, it's kind of a mix of—I think is the full polar ends of the spectrum, right? So there were some people who were like, this is the coolest thing ever, and then there were some people who were like, this is the worst thing that you could buy.

4. 31:34

   Similarly, in terms of the capabilities, there were some people who were like, oh, my God, everybody's going to lose all their jobs tomorrow, which is not what we've ever really believed or thought of this as. To also like, dude, there's no way this is ever going to work, and this is totally a scam. you know so what was the criticism did you did you release the product earlier than you wanted to well look it's it's it wasn't even a product at the time to be honest i mean it was more just like a prototype or like a demo of what was possible and and you know we had been working with it and playing with it for a few months at that point and the kind of just wanted to show people some of the examples of what it was capable of because there was a real like like it was pretty i mean it was pretty insane for us to see as well like i remember the first time that it did

5. 32:17

   like a real task. Like I could not sleep that night. And so it's just like showing that. Wait, wait, say more about that. The first task that Devin did was it like set up MongoDB for us. And it's like, you know, it's a, it's, it's. Standard thing, it's issues that a lot of people run into all the time when they're going and getting their kind of, like, initial, you know, DB set up or whatever. But we would just, like, run into errors.

6. 32:39

   You have this whole flow where you, like, find an error. You Google error. You find some message on Stack Overflow or whatever. Or you ask Chachi to be, and it tells you, okay, here's what you should try. You try that thing. You run into a different error. You should, like, paste that in. And, you know, and, like, for this one, we kind of, like, at some point, We're just like, okay, Devon, just try to go fix it.

7. 33:01

   Just go run commands, like do whatever you need to go do it. And then it worked. And you couldn't sleep. It was truly like, because again, it's like just seeing the exponential curve ahead. Because this was a very specific case. It was the one success that we had. It was very much like definitely a way better than average run. But no, there was this feeling of like, why shouldn't all software and all products be built this way now?

8. 33:28

   Like, you can just tell it what you want it to do and have it go do it. I mean, it's kind of funny because, yeah, to your point, like, we did get hate in the beginning. We did not feel motivated at all from the hate to be like, oh, yeah, no, maybe you guys are right. Like, maybe we should go and, like, you know, focus on the more kind of, like, chatbot Q&A style product experiences. Like, maybe more than we should have.

9. 33:48

   Maybe we should have done something in the middle, you know? I don't know. But, like, I think for us, like, when we had seen that and when we had done these different things, like, all of those, like, demos that we showed, you know, in our launch announcement were like actual runs of Devon that we had done ourselves and like run into and been like, holy shit, this is insane. For us, it was always kind of like, look, you can, you know, we can debate when or like what level of effectiveness or whatever, but like it's just, it's going to happen and that's how we always felt about it.

10. 34:16

    So explain the process of iteration to go from that product, you're getting a lot of hate. Actually, you know what? I called Jeremy Stern who wrote this excellent profile of you in Colossus, which there's a lot of parts I was just laughing my ass off, by the way. And I was asking, I was like, tell me the stuff that didn't make the profile. Yeah. And he said there wasn't that much because you're kind of like an open book and a lot of people like manage their media.

11. 34:39

    Yeah. And, you know, you could have to talk about certain stuff off the record or whatever and you were just like saying everything. But you did say something about like you made the point where like when you released the first product, what was the benchmark? Sweet Bench. And it was at like 13%. That was Devin. Yeah, yeah. And your point was that that's already better than... Yeah, at the time, the best known was like 3% or 4% or something like that, but obviously, yeah, 13%.

12. 35:00

    It still means you fail, you know, 87% of the time. Was there a pronounced benefit from releasing early like that? For sure, yeah. I mean, so for us, by the way, you know, if you kind of think about this overall AI ecosystem, I mean, a simple way to put it, it's like, dude, we were late by a lot. You know, it's like OpenAI started. Why, though? Google DeepMind or Google Brain, I mean, these are obviously, you know, more than a decade old already.

13. 35:22

    OpenAI started like end of 2015 or even, you know, like the Anthropic or, you know, other that folks talk about in the world, like had been around already for years. Like, we were getting started in early 2024. It was like a year plus out from the ChatGPT launch. A lot of the existing players were already there. You know, the same was true in code specifically. I mean, there was, you know, GitHub Copilot, which people had already, you know, engineers had already used for years.

14. 35:45

    And that was very much the, like, Q&A, like the autocomplete style experience of, you know, working with AI. You know, when you start a company, you kind of have nothing. Like, you have no right to exist is maybe one way to put it. But it's like an interesting truth about the world, which is obviously startups succeed. You know, startups succeed all the time. And then startup versus big company has been played out for, you know, for forever.

15. 36:09

    But like, there's no reason, you know, in terms of resources, in terms of people, in terms of brand awareness, like you have none of the things that, you know, the big guys have. And so from that perspective, you have no right to win, you know, over what they're doing, you know. The reason that you're sometimes able to anyway is if you really, like, you know, plant your flag on the ground and put a stake into, like, what you think the future is, and you run like hell towards that, and if you, like, turn out to be right on some of the core things.

16. 36:44

    And, like, I think we were wrong on a lot of things, to be clear. A lot of things we were, you know, definitely we were early, which is a very fair criticism. I think a lot of the details and the nuances, like, we learned and adjusted over time. But the idea that like you would work with AI as a coworker, you know, rather than as like a tool or a chat bot, I think was, I mean, over the last couple of years has obviously like really grown.

17. 37:06

    And I think it was like very important for us, for our brand, for recruiting, for, you know, customer work, everything for us to be, you know, the first ones that actually planted that flag in the ground and said that.