# OpenAI’s CPO on how AI changes must-have skills, moats, coding, startup playbooks, more | Kevin Weil Markdown copy of https://realalpha.app/v/scsW6_2SPC4/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy. Kevin Weil says fast-improving AI rewards teams that test models with evals and build products for capabilities near the edge. - Show: Lenny's Podcast - Hosts: [Lenny Rachitsky](https://realalpha.app/people/lenny-rachitsky/) - Guest: [Kevin Weil](https://realalpha.app/builders/kevin-weil/) is chief product officer at OpenAI. - Published: Apr 10, 2025 - Length: 1 h 32 m ## The best minutes - [19:13](https://realalpha.app/v/scsW6_2SPC4/?t=1153) [Evals tell you what AI product to build](https://realalpha.app/s/evals-tell-you-what-ai-product-to-build/) (6 min) - [59:20](https://realalpha.app/v/scsW6_2SPC4/?t=3560) [Break AI work into specialized models and tasks](https://realalpha.app/s/break-ai-work-into-specialized-models-and-tasks/) (5 min) - [1:17:59](https://realalpha.app/v/scsW6_2SPC4/?t=4679) [Libra failed by trying to change too much at once](https://realalpha.app/s/libra-failed-by-trying-to-change-too-much-at-once/) (4 min) ## Also worth watching - [26:19](https://realalpha.app/v/scsW6_2SPC4/?t=1579) [OpenAI plans for alignment, not a fixed roadmap](https://realalpha.app/s/openai-plans-for-alignment-not-a-fixed-roadmap/) (6 min) - [36:01](https://realalpha.app/v/scsW6_2SPC4/?t=2161) [AI interfaces can borrow from how people think](https://realalpha.app/s/ai-interfaces-can-borrow-from-how-people-think/) (5 min) - [40:53](https://realalpha.app/v/scsW6_2SPC4/?t=2453) [Chat remains a versatile baseline for AI interfaces](https://realalpha.app/s/chat-remains-a-versatile-baseline-for-ai-interfaces/) (3 min) - [47:59](https://realalpha.app/v/scsW6_2SPC4/?t=2879) [OpenAI hires PMs for agency and ambiguity tolerance](https://realalpha.app/s/openai-hires-pms-for-agency-and-ambiguity-tolerance/) (5 min) - [1:04:40](https://realalpha.app/v/scsW6_2SPC4/?t=3880) [AI tutoring is a major unmet opportunity](https://realalpha.app/s/ai-tutoring-is-a-major-unmet-opportunity/) (3 min) Skip the opening bio and sponsor reads, the recruiting story, the AI optimism and creative-work discussion, and the ten-minute lightning round. ## Full analysis: Kevin Weil says 60% accuracy changes AI product design - A model right 60% of the time calls for a different product than one right 95%, Weil says. - Other guests warn that evals can mislead when prompts, answer keys or tasks are poor. - Our read is to validate tests against real workflows before setting promises or adding model calls. Its 6 sections, what it means for you and what to watch are on [the page](https://realalpha.app/v/scsW6_2SPC4/#analysis), free with an account: - [18:14](https://realalpha.app/v/scsW6_2SPC4/?t=1094) Accuracy thresholds shape what a product can promise - [21:50](https://realalpha.app/v/scsW6_2SPC4/?t=1310) Teams define quality while they build the product - [24:00](https://realalpha.app/v/scsW6_2SPC4/?t=1440) Custom evals can reward the wrong work - [1:00:00](https://realalpha.app/v/scsW6_2SPC4/?t=3600) OpenAI splits complex work across specialized models - [27:05](https://realalpha.app/v/scsW6_2SPC4/?t=1625) Quarterly plans align teams, not launch dates - [49:53](https://realalpha.app/v/scsW6_2SPC4/?t=2993) OpenAI hires PMs to act amid ambiguity ## Summary Kevin Weil is chief product officer at OpenAI, where he oversees development of ChatGPT, enterprise products and the OpenAI API. Before OpenAI, he led product at Twitter, Instagram and Planet, and worked on Facebook’s Libra cryptocurrency project, later called Novi. Weil says AI product work differs from conventional software because model capabilities shift rapidly and performance varies by use case, making custom evals central to deciding what to build. He describes OpenAI’s approach as lightweight planning, empowered teams and iterative deployment, and says startups can find opportunities in industry-specific AI products. He argues that chat is a versatile interface and expects teams to build with specialized, fine-tuned model ensembles. He sees potential for AI to expand tutoring and creative work, while saying society must support people affected by the transition. He calls Libra a career disappointment and says the project tried to introduce too many changes at once. ### Minute by minute - [5:13](https://realalpha.app/v/scsW6_2SPC4/?t=313) **Internal buzz predicted image demand.** Weil says OpenAI had a strong internal signal that its image model would catch on: employees used it constantly before launch. He says the Ghibli-style trend was not a planned example, but an illustration of the model’s ability to follow complex instructions, such as arranging items in a room. - [8:14](https://realalpha.app/v/scsW6_2SPC4/?t=494) **AI breakthroughs quickly become ordinary.** Weil says people often call a technology AI while it is still new, then take it for granted once it works reliably. He compares that shift to self-driving cars and GPT-3, which once felt extraordinary but would seem limited to users today. - [11:38](https://realalpha.app/v/scsW6_2SPC4/?t=698) **OpenAI's pace reshaped product work.** Weil had planned to spend the summer with his children after leaving Planet, but Sam Altman recruited him to OpenAI. The interview process moved quickly, then Weil waited nine days for a response because of internal activity. At OpenAI, he found product work shaped by model capabilities changing every couple of months. - [18:37](https://realalpha.app/v/scsW6_2SPC4/?t=1117) **Evals define product readiness.** Weil describes evals as tests that measure how well a model handles a task, such as coding or science. For Deep Research, OpenAI created evaluations around important use cases and used them to track whether the model improved. He says a product must be designed differently depending on whether a model succeeds 60%, 95% or 99.5% of the time. - [24:36](https://realalpha.app/v/scsW6_2SPC4/?t=1476) **Startups can build industry-specific AI.** Weil says OpenAI cannot build products for every industry and use case, and that many applications depend on data held inside companies. He points to the OpenAI API, which he said had 3 million developers using it, as a way to help outside teams build those products. - [26:25](https://realalpha.app/v/scsW6_2SPC4/?t=1585) **Fast iteration supports product leadership.** Weil says OpenAI uses quarterly planning to align teams but expects plans to change as the technology evolves. He favors empowered, bottoms-up teams and shipping early so the company and users can learn together. He also says teams should build near a model’s capability edge, because improvements in a few months may make a barely working product succeed; he credits moving quickly and offering broad capabilities with helping ChatGPT gain consumer awareness. - [36:01](https://realalpha.app/v/scsW6_2SPC4/?t=2161) **Chat remains a flexible AI interface.** Weil says product teams can design model experiences by considering how a person would handle the same situation. For reasoning models, he says OpenAI chose brief progress updates rather than silence or exposing every thought. He argues chat is a versatile way to communicate with AI, while more prescribed interfaces can work better for specific, high-volume tasks. - [45:22](https://realalpha.app/v/scsW6_2SPC4/?t=2722) **Research and product teams must build together.** Weil says OpenAI is working to bring research, engineering, design and product together to build and improve models around real use cases. He estimates the company has about 25 product managers and prefers a PM-light structure. He says its PMs need high agency, comfort with ambiguity and the ability to lead through influence. - [52:59](https://realalpha.app/v/scsW6_2SPC4/?t=3179) **Vibe coding changes prototyping.** Weil says OpenAI staff use ChatGPT to summarize and write documents, product specs and evals, but he thinks they should use AI more. He describes vibe coding as guiding tools such as Cursor or Windsurf with prompts, letting them generate a prototype and correcting mistakes as they appear. He says this can help teams explore ideas and make demos without relying only on designs. - [57:24](https://realalpha.app/v/scsW6_2SPC4/?t=3444) **Specialized models work as an ensemble.** Weil expects researchers and machine-learning specialists to become part of more product teams as fine-tuning becomes routine. He says OpenAI combines models of different sizes, prompts and specialties to solve larger problems. For customer support, he says models can answer many questions automatically and send uncertain cases to a human whose answer can help improve the system. - [1:04:30](https://realalpha.app/v/scsW6_2SPC4/?t=3870) **AI can expand learning and opportunity.** Weil says he and his wife encourage their children to be curious, independent and self-confident. He sees personalized tutoring as one of AI’s important opportunities, and says studies show tutoring alongside classroom education can improve learning. He is optimistic about technology’s long-term effects but says society must support people affected by disruption. - [1:09:56](https://realalpha.app/v/scsW6_2SPC4/?t=4196) **AI broadens creative exploration.** Weil says image generation lets people with limited artistic skills explore ideas, and describes a film director using Sora to brainstorm 50 versions of a scene before taking one to a effects studio for final production. He says models are getting smarter, faster, cheaper and safer, and argues that builders should target capabilities that are nearly possible because models will keep improving. - [1:17:59](https://realalpha.app/v/scsW6_2SPC4/?t=4679) **Weil calls Libra his biggest career disappointment.** Weil says Libra aimed to let people send money through WhatsApp as simply as a text, addressing remittance fees he said could reach 20%. He says the project tried to introduce a new blockchain, a basket of currencies and Messenger and WhatsApp integration at once, while Facebook’s reputation was at a low point. He regrets that the product did not ship, though he says its open-sourced technology lives on in Aptos and Mysten. - [1:21:43](https://realalpha.app/v/scsW6_2SPC4/?t=4903) **Examples improve prompts.** Weil says people should not need to become expert prompt engineers as AI products improve. For now, he recommends giving a model examples of the task and a good answer, which he says can guide it more effectively. He also says assigning the model a role, such as an expert marketer, can shift its response. ## Transcript Lenny's Podcast, hosted by Lenny Rachitsky, with Kevin Weil. Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there. [0:00](https://realalpha.app/v/scsW6_2SPC4/?t=0) The AI models that you're using today is the worst AI model you will ever use for the rest of your life. And when you actually get that in your head, it's kind of wild. Everywhere I've ever worked before this, you kind of know what technology you're building on. But that's not true at all with AI. Every two months, computers can do something they've never been able to do before, and you need to completely think differently about what you're doing. You're chief product officer of maybe the most important company in the world right now. I want to chat about what it's just like to be inside the center of the storm. [0:29](https://realalpha.app/v/scsW6_2SPC4/?t=29) our general mindset is in two months, there's going to be a better model and it's going to blow away whatever the current set of limitations are. And we say this to developers too. If you're building and the product that you're building is kind of right on the edge of the capabilities of the models, keep going because you're doing something right. Give it another couple months and the models are going to be great. And suddenly the product that you have that just barely worked is really going to sing. Famously, you led this project at Facebook called Libra. Libra is probably the biggest disappointment of my career. [0:59](https://realalpha.app/v/scsW6_2SPC4/?t=59) It fundamentally disappoints me that this doesn't exist in the world today because the world would be a better place if we'd been able to ship that product. We tried to launch a new blockchain. It was a basket of currencies originally. It was integration into WhatsApp and Messenger. I would be able to send you 50 cents in WhatsApp for free. It should exist. To be honest, the current administration is super friendly to crypto. Facebook's reputation is in a very different place. Maybe they should go build it now. Today, my guest is Kevin Wheel. Kevin is Chief Product Officer at OpenAI, which is maybe the most important and most impactful company in the world right now, being at the forefront of AI and AGI and maybe someday superintelligence. He was previously head of product at Instagram and Twitter. He was co-creator of the Libra cryptocurrency at Facebook, which we chat about. He's also on the boards of Planet and Strava and the Black Product Managers Network and The Nature Conservancy, he's also just a really good guy, and he has so much wisdom to share. [1:59](https://realalpha.app/v/scsW6_2SPC4/?t=119) We chat about how OpenAI operates, implications of AI and how we will all work and build product, which markets within the AI ecosystem companies like OpenAI won't likely go after and thus are good places for startups to own, also why learning the craft of writing evals is quickly becoming a core skill for product builders, what skills will matter most in an AI era and what he's teaching his kids to focus on, and so much more. This is a very special episode, and I am so excited to bring it to you. If you enjoy this podcast, don't forget to subscribe and follow it in your favorite podcasting app or YouTube. If you become an annual subscriber of my newsletter, you get a year free of Perplexity Pro, Linear, Notion, Superhuman, and Renola. Check it out at lenny'snewsletter.com and click Bundle. With that, I bring you Kevin Wheel. This episode is brought to you by Epo. [2:50](https://realalpha.app/v/scsW6_2SPC4/?t=170) Epo is a next-generation A-B testing and feature management platform built by alums of Airbnb and Snowflake for modern growth teams. Companies like Twitch, Miro, ClickUp, and DraftKings rely on Epo to power their experiments. Experimentation is increasingly essential for driving growth and for understanding the performance of new features. And Epo helps you increase experimentation velocity while unlocking rigorous, deep analysis in a way that no other commercial tool does. When I was at Airbnb, one of the things that I loved most was our experimentation platform, where I could set up experiments easily, troubleshoot issues, and analyze performance all on my own. Epo does all that and more, with advanced statistical methods that can help you shave weeks off experiment time, an accessible UI for diving deeper into performance, and out-of-the-box reporting that helps you avoid annoying, prolonged analytic cycles. Epo also makes it easy for you to share experiment insights with your team sparking new ideas for the a b testing flywheel epo powers experimentation across every use case including product growth machine learning monetization and email marketing check out epo at getepo.com slash lenny and 10x [3:59](https://realalpha.app/v/scsW6_2SPC4/?t=239) Your experiment velocity. That's get EPPO.com slash Lenny. This episode is brought to you by Persona, the adaptable identity platform that helps businesses fight fraud, meet compliance requirements, and build trust. While you're listening to this right now, how do you know that you're really listening to me, Lenny? These days, it's easier than ever for fraudsters to steal PII, faces, and identities. That's where Persona comes in. Persona helps leading companies like LinkedIn, Etsy, and Twilio securely verify individuals and businesses across the world. What sets Persona apart is its configurability. Every company has different needs depending on its industry, use cases, risk tolerance, and user demographics. That's why Persona offers flexible building blocks that allow you to build tailored collection and verification flows that maximize conversion while minimizing risk. Plus, Persona's orchestration tools automate your identity process so that you can fight rapidly shifting fraud and meet new waves of regulation. [5:02](https://realalpha.app/v/scsW6_2SPC4/?t=302) Whether you're a startup or an enterprise business, Persona has a plan for you. Learn more at withpersona.com slash Lenny. Again, that's withpersona.com slash Lenny. Kevin, thank you so much for being here and welcome to the podcast. Thank you so much for having me. we've been talking about doing this forever and we made it happen we did it i can't imagine how insane your life is so i really appreciate you that you made time for this and we're actually recording this uh the week that you guys launched your new image model which is a happy coincidence uh my entire social feed is filled with skibly vacations of everyone's life and family photos and everything so good job yep mine too my wife elizabeth sent me one of hers so i'm i'm right there with you uh let me just ask did you guys expect this kind of reaction it feels like this is the most viral thing that's happened in ai which is a high bar uh since i don't know chat gpt launched just like did you guys expect it to go this well uh what does it feel like internally you know there have been a handful of times in my career when you're working on a project or product internally and the internal usage just explodes uh this was true by the way when we [6:13](https://realalpha.app/v/scsW6_2SPC4/?t=373) were building stories at instagram more than anything else in my career we could feel it was going to work because we were all using it internally and we'd go away for a weekend you know before it launched we were all using it and we'd come back after a weekend and we would know what was going on and be like oh hey i saw you were at that camping trip that how was that you were like man this thing really works image gen was definitely one of those uh so we've been playing with it for i don't know a couple months and um uh it when it first went live internally to the company there was kind of a a little gallery where you could generate your own you could also see what everyone else was generating and it was just like non-stop buzz so yeah we had a sense that this was going to be a lot of fun for people to play with that's a really cool like that should be a measure of just like uh confidence in the something going well that you're launching is internally everyone's going crazy for it yeah especially social things because yeah um you have a very tight network as a company socially so you know each other and your experts and your product hopefully and so there's some sense in which if you're doing something social and it's not taking [7:19](https://realalpha.app/v/scsW6_2SPC4/?t=439) off internally you might you might question what you're doing yeah uh and by the way the ghibli thing is that something you see did or how did that even start was that like an intentional example i think it's just a style people love and model is is really capable at at emulating style or understanding what you know it's very good at instruction following that's actually something that i think people i'm starting to see people discover with it but you do very complex things you can give it two images you know one is your living room and the other is a whole bunch of photos or memorabilia or things you want and you say like tell me how you would arrange these things or you can say i'd like you to show me what this will look like if you put this over here and this thing to the right of that and this one to the left of this, but under that one. And the model actually will understand all of that and do it. It's incredibly powerful. So I'm excited about all the different things people are going to figure out. Yeah. All right. Well, good job. Good job, team OpenAI. Let's get serious here and let's kind of zoom out a little bit. The way I see it is your chief product officer of maybe the most important company in the world right now. Just not to set the bar too high, but you guys [8:24](https://realalpha.app/v/scsW6_2SPC4/?t=504) are ushering in ai agi at some point super intelligence at some point no big deal uh i've had i have more questions for you than i've had for any other guest actually put out a call out on twitter and linkedin and my community just like what would you want to ask kevin and i had 300 over 300 well-formed questions and we're going to go through every single one so let's just get started i'm just joking i picked out the best and there's a lot of stuff i'm really curious it's it's 1 p.m here it doesn't get dark for a while so let's do it okay here we go okay Okay, so first of all, I'm just going to take notes here. [8:55](https://realalpha.app/v/scsW6_2SPC4/?t=535) When is AGI launching? When is this? I mean, we just launched a good image gen model. Does that count? It's getting there. It's getting there. There's this quote I love, which is, AI is whatever hasn't been done yet. Because once it's been done, when it kind of works, then you call it machine learning. And once it's kind of ubiquitous and it's everywhere, then it's just an algorithm. So I've always loved that. that we call things ai when they still don't quite work and then you know by the time it's like an ai algorithm that's recommending you follow you know oh that's just an algorithm but this new thing like self-driving cars that's it to some degree we're always going to be there and the next thing is always going to be ai and the current thing that we you know use every day and is just a part of our lives that's an algorithm it's so interesting because yeah like uh in the bay area you see self-driving cars driving around it's so normal now when like four years ago i don't know three years ago you would have thought you would have seen this and you'd be like holy shit what is how we're in the future and now we're just so taken for granted it's i mean there's something like that with everything if i showed you it when gpt3 launched right i wasn't at open [10:08](https://realalpha.app/v/scsW6_2SPC4/?t=608) ai then i was just uh i was just a user but it was mind-blowing and if i gave you gpt3 now i just plug that into chat gpt for you and you started using it you'd be like what is this thing i like it's like mess uh there's i had the same experience when i when i first got into a waymo right your your very first ride at least my very first ride my first like 10 seconds in a waymo it starts driving and you're like oh my god watch out for that bike you're holding on to whatever you can and then like five minutes in you've calmed down and you realize that you're getting driven around the city without a driver and it's working you're just like oh my god i am living in the future right now and then like another 10 minutes you're bored you're doing email on your phone answering slack messages and you know suddenly this miracle of human invention is just an expected part of your life from then on and there is really something in the way that we all are adapting to ai that's kind of like that these miraculous things happen and computers can do something they've never been able to do before and it blows our mind collectively for like a week and then we're like oh yeah like oh yeah now it's just machine learning on its way to being an [11:22](https://realalpha.app/v/scsW6_2SPC4/?t=682) algorithm the craziest thing about what you just shared actually is like i don't know chat gpt which is like now feels terrible uh 3.5 was like a couple years ago and uh imagine what life will be like in a couple years from now we're gonna get to that where things are going what you think is going to be the next big leap. But I want to start with the beginning of your journey at OpenAI. So you worked at Twitter, you worked at Facebook, you worked at Planet, Instagram. At some point, you got recruited to go and come work at OpenAI. I'm curious just what that story was like of the recruiting process of joining OpenAI as CPO. Are there any fun stories there? [12:00](https://realalpha.app/v/scsW6_2SPC4/?t=720) If I'm remembering the timeline right, we communicated a planet. I was leaving, and I was planning to just go take some time. I wasn't going to stop working, but I was also happy to take the summer. This is maybe April or something. I was like, cool, I'm going to have the summer with my kids. We're going to go to Tahoe or something, and I'll actually get to hang out rather than what I usually do going up and down and all that. and then you know sam and i had known each other lightly for a bunch of years and he's he's always involved in so many interesting things you know like companies building fusion and and all these things so he'd always been somebody that i would like call occasionally if i was starting to think about my next thing um because i like working on big like tech forward sort of you know next next wave kind of things and um and so uh i called him i think vinod also helped uh put us in touch again and and this time it wasn't like oh you should go talk to like these guys working on fusion it he said actually you know that we're thinking about something you should come talk to us i was like okay that sounds amazing let's do it and it goes really fast really really fast like i [13:12](https://realalpha.app/v/scsW6_2SPC4/?t=792) met uh you know most of the management team in a brief period of time a few days and they were telling me look we're gonna we're basically gonna move as fast as we as we want to move and uh kind of if everyone if you talk to everyone everyone likes you're ready to go uh sam came over for dinner uh and we had we had a great evening together just like talking about open AI in the future and getting to know each other better. And at the end, I was like, I was going to go in the next day for like a bigger round of interviews. And Sam was saying, you know, hey, it's going really well. We're really excited. And I said, cool. So how do I think about tomorrow? [13:53](https://realalpha.app/v/scsW6_2SPC4/?t=833) And he said, oh, you'll be fine. Don't worry about it. And if it goes well, like we're basically there. And so I go in the next day, meet a bunch of people, have a great time. Like I really enjoyed everybody I met with in any interview you can always second guess yourself you know like oh I shouldn't have said that thing or I that thing I gave a bad answer on I wish I could redo but I came away feeling like I think that went pretty well and I was expecting to hear like that weekend basically because they'd sort of set expectations as soon as you know if this goes well we're ready to go and uh i didn't hear anything and then it was like monday tuesday wednesday i still didn't hear anything and uh i reached out to uh to folks on the opening eye side a couple of times still nothing and i was like oh my god i screwed it up like i don't know where i screwed it up but i totally screwed it up i can't believe it and i was going back to elizabeth my wife and being like what did i do like where where do you think i you know getting all crazy about it and um and then it's still nothing and finally it was like it was like nine days later they finally got back to me and it turned out you know there was like a bunch of stuff happening internally [15:10](https://realalpha.app/v/scsW6_2SPC4/?t=910) and this that and the other thing and uh you know there's just a million things happening and they finally were like oh yeah that went well let's do this and i was like oh okay cool let's do it. Uh, but, uh, it was like nine days of agony and they were just super busy on some internal stuff. And, uh, there I was like fretting every single day and re re going over every line of our interview process. It makes me think about when you're like dating someone and you would text them and they just, you're not hearing anything back and all like you assume something is wrong. Yeah, totally. They might just be busy. Uh, I give them a hard time about it still. [15:44](https://realalpha.app/v/scsW6_2SPC4/?t=944) so uh that's wild uh i love i love that it worked out uh and i guess i guess the lesson there is don't don't jump to conclusions yeah and have a little bit of chill speaking of that I want to chat about what it's just like to be inside the center of the storm. Again, you worked at a lot of, let's say, traditional companies, even though they're not that traditional, Twitter and Instagram and Facebook and Planet, and now you work at OpenAI. I'm curious what is most different about how things work in your day-to-day life at OpenAI. I think it's probably the pace. Maybe it's two things. One is it's the pace. [16:24](https://realalpha.app/v/scsW6_2SPC4/?t=984) The second is, you know, everywhere I've ever worked before this, you kind of know what technology you're building on. So you spend your time thinking about what problems are you solving? Who are you building for? How are you going to make their lives better? Is this a big enough problem that you're going to be able to change habits? Do people care about this problem being solved? All those good product things. But the stuff that you're building on is kind of fixed. You're talking about databases and things. And I bet the database you used this year is probably 5% better than the database you used two years ago. But that's not true at all with AI. It's like every two months, computers can do something they've never been able to do before. And you need to completely think differently about what you're doing. There's like something fundamentally interesting about that makes life fun here. There's also something, you know, we'll maybe like talk about evals later, but that it also really in this world of everything we're used to with computers is about giving a computer very defined inputs. You know, if you look at Instagram, for example, there are buttons that do specific things and you know what they do. [17:36](https://realalpha.app/v/scsW6_2SPC4/?t=1056) And then when you give a computer defined inputs, you get very defined outputs. You're confident that if you do the same thing three times, you're going to get the same output three times. LLMs are completely different than that, right? They're good at fuzzy, subtle inputs, then all the nuances of human language and communication, they're pretty good at. and also they don't really give you the same answer you you probably get spiritually the same answer for the same question but it's certainly not the same set of words every time and so you're much more it's fuzzier inputs and fuzzier outputs and it when you're building products it really matters with whether you know there's some use case that you're trying to build around if the model gets it right 60 percent of the time you build a very different product than if the model gets it right 95% of the time versus if the model gets it right 99.5% of the time. [18:29](https://realalpha.app/v/scsW6_2SPC4/?t=1109) And so there's also something that you have to get really into the weeds on your use case and the evals and things like that in order to understand the right kind of product to build. So that is just fundamentally different. If your database works once, it works every time. And that's not true in this world. Let's actually follow this thread on evals. I definitely wanted to talk about this. So we had this legendary panel at the Lending Friends Summit it was you and Mike Krieger and Sir Guo moderating. So fun. And the thing that I heard that kind of stuck with people from that panel was a comment you made where you said that writing evals is going to become a core skill for product managers. And I feel like that probably applies further than just product managers. A lot of people know what evals are. A lot of people have no idea what I'm talking about. So could you just briefly explain what is an eval? And then just why do you think this is going to be so important for people building products in the future? [19:22](https://realalpha.app/v/scsW6_2SPC4/?t=1162) Yeah, sure. I think the easiest way to think about it is almost like a quiz for a model, a test to gauge how well it knows a certain set of subject material or how good it is at responding to a certain set of questions. So in the same way, you take a calculus class and then you have calculus tests that see if you've learned what you're supposed to learn. You have evals that test how good is the model at creative writing? How good is the model at graduate level science? how good is the model at competitive coding and so you have these set of evals that basically perform as benchmarks for how smart or capable the model is. [20:04](https://realalpha.app/v/scsW6_2SPC4/?t=1204) Is it like a simple way to think about it, like unit tests for model? Yeah, unit tests, tests in general for models, totally. Great, great. Okay, and then why is this so important for people that don't totally understand what the hell's going on here with evals? Why is this so key to building AI products? Well, it gets back to what I was saying. You need to know whether your model is going to, there are certain things that models will get right 99.95% of the time, and you can just be confident. it. There are things that they're going to be 95% right on and things they're going to be 60% right on. If the model is 60% right on something, you're going to need to build your product totally differently. And by the way, these things aren't static either. So a big part of evals is if you know you're building for some use case. So let's say, let's take our deep research product, which is one of my favorite things that we've released maybe ever. The idea is with deep research for people who haven't used it you can give chat gpt now a an arbitrarily complex query like it's not about returning you an answer from you know a search query which we can also do it's it's here's a thing that if you were going to answer it yourself you'd go off and do you know two hours [21:15](https://realalpha.app/v/scsW6_2SPC4/?t=1275) of reading on the web and then you might need to read some papers and then you would come back and start writing up your thoughts and realize you had some gaps in your thinking so you go out and do more research you might it might take you a week to write some like 20 page answer to this question you can let chat gpt just like chug for you for 25 30 minutes you know it's not the immediate answers you're used to but it might go work for 25 30 minutes and do work that would have taken you a week so as we were building that product we were designing evals uh sort of it at the same time as we were thinking about how this product was going to work and we were trying to go through like hero use cases you know here's a question you want to be able to ask here's an amazing answer for that question and and then turning those into evals and and then hill climbing on those evals so it's not just that the model is static and we hope it does okay on a certain set of things you can teach the model you can make this a continuous learning process and so as we were fine tuning our model for deep research to be able to answer these things, we were able to test, is it getting better on these evals that we said were important measures of how the product [22:27](https://realalpha.app/v/scsW6_2SPC4/?t=1347) **Kevin Weil:** was working? And it's when you start seeing that and you start seeing performance on evals going up, you start saying, okay, I think we have a product here. You made a kind of a comment along [22:36](https://realalpha.app/v/scsW6_2SPC4/?t=1356) these same lines around evals that AI is almost like capped in how amazing it can be by how good we are at eBells. Does that resonate? Any more thoughts along those lines? I mean, these models are, they're intelligences, and intelligence is so fundamentally multidimensional. So you can talk about a model being amazing at competitive coding, which may not be the same as that model being great at front-end coding, or back-end coding, or taking a whole bunch of code that's written in COBOL and turning it into Python, you know? And that's just within the software engineering world. And so I think there's a sense in which you can think of these models as incredibly smart, very factually aware intelligences, but still most of the world's data knowledge process is not public. It's behind the walls of companies or governments or other things. And same way, if you were going to join a company, you would spend your first two weeks onboarding you'd be learning the company specific processes you'd get access to company specific data it's you can teach these the models are smart enough you can teach them anything but they need to have the sort of the raw data uh to to learn from and so there's a there's a sense [23:59](https://realalpha.app/v/scsW6_2SPC4/?t=1439) in which Yeah, I think the future is really going to be incredibly smart, broad-based models that are fine-tuned and tailored with company-specific or use-case-specific data so that they perform really well on company-specific or use-case-specific things. And you're going to measure that with custom evals. And so, you know, what I was referring to is just like these models are really smart. you need to still teach them things if the data is not in their training set. And there's a huge amount of use cases that are not going to be in their training set because they're relevant to one industry or one company. I'm just going to keep following the thread that you're leading us down, but I'm going to come back because I want more questions around some of these things. [24:45](https://realalpha.app/v/scsW6_2SPC4/?t=1485) So you came to a space that I think a lot of AI founders are thinking about is just where is OpenAI not going to come squash me in the future or one of the other foundational models? And so it's unclear to a lot of people just like, should I build a startup in the space or not? Is there any advice you have or any guidance for where you think OpenAI or just foundational models in general likely won't go and where you have an opportunity to build a company? Well, one of my, so this is something that Ev Williams used to say back at Twitter that's always stuck with me, which is no matter how big your company gets, no matter how incredible the people are, there are way more smart people outside your walls than there are inside your walls. [25:26](https://realalpha.app/v/scsW6_2SPC4/?t=1526) And that's why we are so focused on building a great API. We have 3 million developers using our API. No matter how ambitious we are, how big we grow, by the way, we don't want to grow super big. There are going to be, there are so many use cases, places in the world where AI can fundamentally make our lives better. We're not going to have the people, we're not going to have the the you know the the know-how to build most of these things and i think like i was saying the data is is industry specific use case specific you know behind certain company walls things like that and there are immense opportunities in every industry in every vertical in the world to go build ai based products that improve upon the the state of the art and there's just no way we could ever do that ourselves we don't want to we couldn't if we didn't want to and we're really excited to power that for 3 million plus developers and way more in the future. Coming back to your earlier point about the tech changing constantly and getting faster, not exactly knowing what you'll have by the time you launch something in terms of the power of the model. I'm curious what allows you to ship quickly and consistently and such great stuff. And it sounds like one answer is [26:43](https://realalpha.app/v/scsW6_2SPC4/?t=1603) bottoms up, empowered teams versus a very top down roadmap that's planned out for a quarter. What what are some of those things that allow you to ship such great stuff so often, so quickly? Yeah, I mean, we try and we try and have a sense of where we're trying to go, you know, point ourselves in a direction so that we have some rough sense of alignment. Like thematically, I don't for a second and we do quarterly roadmapping. You know, we laid out sort of a year long strategy. [27:12](https://realalpha.app/v/scsW6_2SPC4/?t=1632) **Kevin Weil:** i don't for a second believe that what we write down in these documents is what we're going to actually ship you know three months from now let alone six or nine but that's okay there's a um i [27:22](https://realalpha.app/v/scsW6_2SPC4/?t=1642) think it's like an eisenhower quote plans are useless planning is helpful uh which i totally subscribe to especially in this world it's really valuable if you think about quarterly roadmapping for example it's really valuable to have a moment where you stop and go okay what did we do what worked what went well what didn't go well what did we learn and now what do we think we're going to do next and by the way everybody has some dependencies you you know you need the infrastructure team to do the following things partnership with research here and so you want to have a second to kind of check your dependencies make sure you're good to go and then start executing we try and keep that really lightweight because Thank you. [28:00](https://realalpha.app/v/scsW6_2SPC4/?t=1680) Because it's not going to be right. You know, we're going to throw it out halfway because we will have learned new things. So the moment of planning is helpful, even if you're only going to, you know, it's only partially right. So that's, I think, just expecting that you're going to be super agile, and that there's no sense writing a three month roadmap, let alone a year long roadmap, because the technology is changing underneath you so quickly. We really do try and go like very strongly bottoms up, kind of subject to our overall directional. alignment. We have great people. We have engineers and PMs and designers and researchers who are passionate about the products they're building and have strong opinions about them and are also the ones building them. They have a real sense of what the capabilities are too, which is super important. I think you want to be more bottoms up in this way. We operate that way. We are happy making mistakes we make mistakes all the time it's one of the things i really appreciate about sam he pushes us really hard to move fast but he also understands that with moving fast comes uh we didn't quite get this right or you know that we launched this thing it didn't work we'll [29:10](https://realalpha.app/v/scsW6_2SPC4/?t=1750) roll it back you know look at our naming our naming is horrible there's a lot of questions people had for you yeah model names yeah it's absolutely atrocious and we know it um and we'll get around to fixing it at some point but it's not the most important thing and so we don't spend a lot of time on it but it also shows you how it doesn't matter uh again chat gpt the most popular fastest growing product in history uh models are it's the number one ai api and model so clearly it doesn't matter that much and we name things like oh three mini high oh man i love it um okay so you talked about road mapping um and bottoms up and i'm really curious how you is there like a cadence or ritual of aligning with you or sam or he or you review everything that's going out like is there a meeting every week or every month where you guys see what's happening on key projects so we do product reviews and things like that like you would expect um there isn't a ritual because there isn't uh we we i would never want us to be blocked on launching something you know waiting for a review with me or sam if we can't get there if i'm traveling or sam's you know busy or whatever that's a bad reason for us not to ship [30:24](https://realalpha.app/v/scsW6_2SPC4/?t=1824) so obviously for the biggest most high priority stuff we have a pretty close beat on it but we really try not to frankly like we want to empower teams to move quickly and uh i think it's more important to ship and iterate so we have this philosophy that we call iterative deployment and the idea is like we're all learning about these models together so there's a real sense in which it's way better to like ship something even when you don't know the full set of capabilities and iterate together like in public and we kind of co-evolve together with the rest of society as we learn about these things and where they're different and where they're good and bad and weird i really like that philosophy um there's also a bit of i think the other thing that ends up being a part of our product philosophy is the sense of model maximalism. [31:21](https://realalpha.app/v/scsW6_2SPC4/?t=1881) The models are not perfect. They're going to make mistakes. You could spend a lot of time building all kinds of different scaffolding around them. And by the way, sometimes we do because sometimes there are things, kinds of errors that you just don't want to make. But we don't spend that much time building scaffolding around the parts that don't match that. Because our general mindset is in two months, there's going to be a better model and it's going to blow away whatever the current set of limitations are. And so if you're building, and we say this to developers too, if you're building and the product that you're building is kind of right on the edge. [32:00](https://realalpha.app/v/scsW6_2SPC4/?t=1920) Edge of the capabilities of the models. Keep going, because you're doing something right. Because you give it another couple months, and the models are going to be great. And suddenly, the product that you have that just barely worked is really going to sing. And, you know, that's, that's kind of how you make sure that you're really pushing the envelope and building new things. I had the founder of Bolt on the podcast StackBlitz is the company name and he shared the story that they've been working on this product for seven years behind the scenes and it was failing, nothing was happening and then all of a sudden it was, sorry to mention a competitor but Claude came out or Sonnet 3.5 came out and all of a sudden everything worked and they've been building all this time and finally it worked and I hear that a lot with YC just like things that never were possible now are just becoming possible every few months with the updates to the models. [32:48](https://realalpha.app/v/scsW6_2SPC4/?t=1968) Yeah, absolutely. Let me actually ask this. I wasn't planning to ask this, but I'm curious if you have any quick thoughts. Just why is Sonnet so good at coding and kind of thoughts on your stuff getting as good and better at actual coding? Yeah, I mean, kudos to Anthropic. They've built very good coding models, no doubt. We think that we can do the same. Maybe by the time this podcast is shipped, we'll have more to say. but either way uh all credit to them i think this intelligence is really multi-dimensional and so i think there's the the model providers it used to be that open ai had this like massive model lead you know 12 months or something ahead of everybody else that's not true anymore you know i like to think we still have a lead i'd argue that we do but it's certainly not a massive one and that means that there are going to be different places where you know the google models are really good or where Anthropics models are really good or where we're really good and our competitors are like, ah, we got to get better at that. [33:51](https://realalpha.app/v/scsW6_2SPC4/?t=2031) And it actually is easier to get better at a certain thing once someone's proved it possible than it is to forge a path through the jungle and doing something brand new. So I just think, yeah, as an example, it was like nobody could break four minutes in the mile and then finally somebody did and the next year 12 more people did it. I think there's that all over the place. And it just means that competition is really intense and consumers are going to win and developers are going to win and businesses are going to win in a big way from that. It's part of why the industry moves so fast. But, you know, all respect to the other big model providers, models are getting really good. [34:32](https://realalpha.app/v/scsW6_2SPC4/?t=2072) We're going to move as fast as we can. And I think we've got some good stuff coming. Exciting. This makes me also think about, in many ways, other models are better at certain things, But somehow ChatGPT is like the, like if you look at all the awareness numbers and usage numbers, it's like no matter where you guys are in the rankings, people seem to just like think of AI and ChatGPT almost as the same. What do you think you did right to kind of win in the consumer mindset, at least at this point, in awareness in the world? I think being first helps, which is one of the reasons why we're so focused on moving quickly. [35:06](https://realalpha.app/v/scsW6_2SPC4/?t=2106) You know, we like being the first to launch new capabilities, things like deep research. uh we've also our models are very they can do a lot of things right so they can they can take real-time video input they can you have speech to speech you can do speech to text and text to speech um they can do deep research they can operate on a canvas they can write code and so chat gpt can kind of be this one-stop shop where all the things that you want to do are possible um and as we as we go forward in it you know we have more agentic tools like operator where it's browsing for you and doing things for you on the web you know more and more you're going to be able to come to this one place to chat gpt give it instructions and have it accomplish real things for you in the world there's like something fundamentally valuable in that and so you know we think a lot about that we think and it we we move we try to move really fast so that we are always the most useful place for people to come to. [36:03](https://realalpha.app/v/scsW6_2SPC4/?t=2163) What would you say is the most counterintuitive thing that you've learned after building AI products or working at OpenAI? Something that's just like, I did not expect that. I don't know, maybe I should have expected this, but one of the things that's been funny for me is the extent to which you can kind of reason, when you're trying to figure out how some products should work with AI, you can often or even why some ai thing happens to be true you can often reason about it the way you would reason about another human and it kind of works yeah so maybe a couple examples when we were first launching our um our reasoning model right we were the first to to build a model that could reason that could that could instead of giving you just a quick you know system one answer right away at every question you asked it was the third emperor of the holy roman empire like you know here's an answer you could ask it hard questions and it would reason the same way that if i asked you to do a crossword puzzle you couldn't just like snap fill in everything you would be well okay on this one across i think it could be one of these two but that means there's an a here so that one has to be this oh wait you know like backtrack kind of step by step build up [37:16](https://realalpha.app/v/scsW6_2SPC4/?t=2236) from where you are. Same way you answer any difficult logistical problem, any scientific problem. So this reasoning breakthrough was big, but it was also the first time that a model needed to sit and think. And that's a weird paradigm for a consumer product. You don't normally have something where you might need to hang out for 25 seconds after you ask a question. And so we were trying to figure out, you know, what's the UI for this? Because it's also not like with deep research where the model is going to go and think for 25 minutes sometimes it's actually not that hard because you're not going to sit and watch it for 25 minutes you're going to go do something else you're going to go to another tab or go get lunch or whatever and then you'll come back and it's done when it's like 20 25 seconds or 10 seconds it's a long experience it's a long time to wait but it's not long enough to go do something else and so you actually need and you know so you [38:09](https://realalpha.app/v/scsW6_2SPC4/?t=2289) **Kevin Weil:** you can think like if you asked me something that i needed to think for 20 seconds to answer what would i do i wouldn't just like go mute and not say anything and kind of um you know shut down [38:23](https://realalpha.app/v/scsW6_2SPC4/?t=2303) for 20 seconds and then come back so we shouldn't do that we shouldn't just like have a slider sitting there that's annoying but i also wouldn't just start like babbling every single thought that i had um so we probably shouldn't just like expose the whole chain of thought as the model's thinking but you know i might go like huh that's a good question all right i might approach it like that and then think and you know you're sort of like maybe giving little updates and that's actually what we ended up shipping you have similar things where you can like you can find situations where um you get better thinking sometimes out of a group of models uh that all try and attack the same problem and then you have a model that's looking at all their outputs and integrating it and then giving you a single answer at the end, I mean, sounds a little bit like brainstorming, right? I certainly have better ideas when I get in a room and brainstorm with other people because they think differently than me. And so anyways, there's just like all these situations where you can actually kind of reason about it like a group of humans or an individual human and it sort of works, which I don't know, maybe I shouldn't have been surprised, but I was. [39:27](https://realalpha.app/v/scsW6_2SPC4/?t=2367) That is so interesting because when I see these models operate, I never even thought about you guys designing that experience like it to me just feels like this is what the llm does it just sits there and tells me what it's thinking and i love this point you're making of like we like let's make it feel like a human operating and how does human operate well they just talk out loud they think here's the thing i should explore and i love that deep sequined like to the extreme of that right where they're just like here's everything i'm doing and thinking and i and people actually like that too i guess was that was that surprising to you like oh maybe that could work too people seem to like everything yeah we learned from that actually um because we um when we first launched it we kind of gave you like the the subheadings of what the model was thinking about but not much more and then deep seek launched and they were it was a lot and we kind of went you know i don't know if everyone wants like that there's some novelty effect to seeing what the model's really thinking about we felt that too when we were looking at it internally it's interesting to see the model's chain of thought but it's not i you know i think at the scale of like 400 million [40:29](https://realalpha.app/v/scsW6_2SPC4/?t=2429) people you don't want to see the model kind of like babble a bunch of things um and so what we ended up doing was summarizing it in interesting ways so instead of just getting the subheadings you're kind of getting like one or two sentences about how it's thinking about it and you can learn from that so we kind of tried to find a middle ground that that we thought was an experience that would be meaningful for most people but you know showing everybody like three paragraphs laughs is probably not the right answer. This reminds me of something else you said at the summit that has really stuck with me. This idea that chat, people always make fun of like chat is not like the future interface for how we interact with AI. But you made this really interesting point that may argue the other side, which is like as humans, we interface by talking and the IQ of a human can span from really low to really high. And it all works because we're talking to them and chat is the same thing and it can work on all kinds of intelligence levels. [41:22](https://realalpha.app/v/scsW6_2SPC4/?t=2482) uh maybe just share maybe i just shared it but uh i guess anything there about just why chat actually ends up being such an interesting interface for all elements yeah i don't know if uh maybe i'm uh maybe this is one of those things i believe that [41:34](https://realalpha.app/v/scsW6_2SPC4/?t=2494) **Kevin Weil:** most people don't believe but i actually think chat is an amazing interface because it's so versatile um people tend to go oh chat yeah well that's just like you know we'll figure out [41:45](https://realalpha.app/v/scsW6_2SPC4/?t=2505) something better and i kind of think i kind of think this is uh it's it's it's incredibly universal because it is the way we talk like i can talk to you verbally like we're talking now i can you know we can see each other and interact uh we can talk on whatsapp and you know be texting each other but all of these things is this sort of like unstructured uh you know method of communication and that's how we operate if i had to and if i had some more rigid interface that i was allowed to use when we spoke, I would be able to speak to you about, you know, far fewer things and it would actually get in the way of us having like maximum communication bandwidth. So there's something magical. And by the way, in the past, it never worked because models there, there wasn't a model that was good at understanding all of the complexity and nuances of human speech. And that's the magic of LLMs. So to me, it's like an interface that's exactly fit to the power of these things. And that doesn't mean that it always has to be just like, I don't necessarily always want to type but if you you do want that very open-ended flexible communication medium it may be that we're speaking and the model's speaking back to me but you still want that like that the very sort of [42:58](https://realalpha.app/v/scsW6_2SPC4/?t=2578) lowest common denominator um no restrictions way of of interacting that is so interesting that's really changed the way i think about the stuff is that point that chat is just so good for this very specific problem of talking to super intelligence basically by the way i think there are like it's not that it's only chat either like there are if you have high volume use cases where they're more prescribed and the you don't actually need the full generality there are there are many use cases where it's better to have something that's less flexible more prescribed faster at a specific task and those are great too and you know you can build all sorts of those and but you still want chat as like this baseline for anything that falls out of whatever, you know, vertical you happen to be building for. [43:46](https://realalpha.app/v/scsW6_2SPC4/?t=2626) It's like a catch-all for like every possible thing you'd ever want to express to a model. I'm excited to chat with Christina Gilbert, the founder of One Schema, one of our longtime podcast sponsors. Hi, Christina. Yes, thank you for having me on, Lenny. The latest with One Schema. I know you now work with some of my favorite companies like Ramp, Vanta, Scale, and Watershed. I heard that you just launched a new product to help product teams import CSVs from especially tricky systems like ERPs. Yes, so we just launched One Schema file feeds, which allows you to build an integration with any system in 15 minutes as long as you can export a CSV to an SFTP folder. We see our customers all the time getting stuck with hacks and workarounds. And the product teams that we work with don't have to turn down prospects because their systems are too hard to integrate with. We allow our customers to offer thousands of integrations without involving their engineering team at all. I can tell you that if my team had to build integrations like this, how nice would it be to be able to take this off my roadmap and instead use something like one schema and not just to build it but also to maintain it forever? [44:48](https://realalpha.app/v/scsW6_2SPC4/?t=2688) Absolutely, Lenny. We've heard so many horror stories of multi-day outages from even just a handful of bad records. We are laser focused on integration reliability to help teams end all of those distractions that come up with integrations. We have a built-in validation layer that stops any bad data from entering your system, and One Schema will notify your team immediately of any data that looks incorrect. I know that importing incorrect data can cause all kinds of pain for your customers and quickly lose their trust. Christina, thank you for joining us, and if you want to learn more, head on over to oneschema.co. That's oneschema.co. [45:23](https://realalpha.app/v/scsW6_2SPC4/?t=2723) I want to come back to that you talked about researchers and their relationship with product teams. I imagine a lot of innovation comes from researchers, just like having an inkling and then building something amazing and then releasing it. And some ideas come from PMs and engineers. How do those teams collaborate? Does every team have a PM? Is it a lot of research-led stuff? Just give us a sense of just where ideas and products come from mostly. it's an area where we're evolving a lot i'm really excited about it frankly i think if you go back you know a couple years when chat gpt was just getting started uh obviously i wasn't an open ai so um but uh it we were more we were more of a pure research company at the time chat gpt if you remember was a low-key research preview um it was for many years yeah it wasn't a thing that the team launched thinking it was going to be this massive product oh chat gpt yeah and it it was just a way that we were going to let people you know play with and iterate on the models um and so we were we were primarily a research company a world-class research company and as chat gpt has grown and as we've built our b2b products and our apis and other things [46:40](https://realalpha.app/v/scsW6_2SPC4/?t=2800) now we're more of a product company than we were i still think we can't we're open ai should never be a pure product company we need to be both a world-class research company and a world-class product company and the two need to really work together and that's the thing that's um that i think we've been getting much better at over the last like six months if you if you treat those things separately and you know the researchers go do amazing things and build models and then they get to some state and then the product and engineering teams go take them and do something with them. We're effectively just an API consumer of our own models. The best products though are going to be, it's like I was talking about with deep research. It's a lot of iterative feedback. [47:22](https://realalpha.app/v/scsW6_2SPC4/?t=2842) It's understanding the products you're trying to solve or the problems you're trying to solve, building evals for them, using those evals to go gather data and fine-tune models to get them to be better at these use cases that you're looking to solve. It's a huge amount of back and forth. uh to do it well and i think the best products are going to be eng product design and research working together as a single team to to build novel things so that's that's actually how we're trying to operate with basically anything that we build it's a new muscle for us because we're kind of new as a product company but um but it's one that people are really excited Because we've seen every time we do it, we build something awesome. And so now every product starts like that. How many product managers do you have at OpenAI? I don't know if you share that number, but if you do. Not that many, actually. I don't know, 25? Maybe it's a little more than that. [48:18](https://realalpha.app/v/scsW6_2SPC4/?t=2898) But my personal belief is that you want to be pretty PM light as an organization just in general. i say this with love because i am a pm but too many pms causes problems you know we'll like fill the world with decks and ideas versus execution so i think that the the i think it's a good thing when you have a pm that has uh that is working with maybe slightly too many engineers because it means that they're not going to get in and micromanage you're going to leave a lot of of influence and responsibility with the engineers to make decisions. It means you want to have really product-focused engineers, which we're fortunate to have. We have an amazingly product focused high agency engineering team. But when you have something like that, you have a team that feels super empowered. You have a PM that's trying to really understand the problems and kind of gently guide the team a little bit, but has too much going on to get too far into the details and you end up being able to move really fast so that's kind of the philosophy we take uh we want we want product the eng leads and and product the engineers all the way through um we want not too many pms but really awesome high quality ones um and so far that seems to [49:34](https://realalpha.app/v/scsW6_2SPC4/?t=2974) be working pretty well i imagine being a pm at open ai is like a dream come true for a lot of people uh at the same time i imagine it's not a fit for a lot of people there's researchers involved very product-minded engineers what do you what do you look for in the pms that you hire there for folks that are like maybe i shouldn't go work there i shouldn't even think about that [49:53](https://realalpha.app/v/scsW6_2SPC4/?t=2993) **Kevin Weil:** i think i've said this a few times but like high agency is something that we really look for people that are not gonna come in and kind of wait for everyone else to allow them to do something they're [50:03](https://realalpha.app/v/scsW6_2SPC4/?t=3003) just gonna see a problem and go do it um that's it's just a core part of how we work i think people that that are happy with ambiguity because there is a massive amount of ambiguity here is not the kind of place and we have we have trouble sometimes with um with more junior pms because of this because it's just not the place where someone is going to come in and say okay you know here's here's the landscape here is your area i want you to go do this thing and that's that's what you want as a as an early career pm we just i mean no one here has time and the problems are too ill-formed and we're figuring them all out as we go. [50:44](https://realalpha.app/v/scsW6_2SPC4/?t=3044) And so high agency, very comfortable with ambiguity, ready to come in and help execute and move really quickly. That's kind of our recipe. And I think also happy leading through influence because, I mean, it's usual as a PM, people don't report to you. Your team doesn't report to you, etc. But you also have the complexity of a research function, which is even more sort of self-directed. And it's really important to build a good rapport with the research team. And so, you know, I think the EQ side of things is also super important for us. I know in most companies, a PM comes in and they're just like, why do we need you? [51:27](https://realalpha.app/v/scsW6_2SPC4/?t=3087) And as a PM, you have to earn trust and help people see the value. And I feel like at OpenAI, it's probably a very extreme version of that where they're like, why do we need this person? We have researchers, engineers. What are you going to do here? Yeah, I think people appreciate it done right. But you bring people along. I think one of the most important things a PM can do well is be decisive. So there's a real fine line. you don't want to be making it i mean it's kind of like i i don't love the pm is the ceo of the product uh illusion all the time but but just like sam and his role would be making mistakes if he made every single decision in every meeting that he was in and he would also be making mistakes if he made no decisions in any meetings that he was in right it's a it's the it's understanding when to defer to your team and like let let people innovate and when there is like a decision to be made that people either don't feel comfortable with or don't feel empowered to make or a decision that, you know, has too many different like disparate pros and cons that are spread out across a big group and someone needs to be decisive and make a call. It's a really important trait of a CEO. It's [52:41](https://realalpha.app/v/scsW6_2SPC4/?t=3161) something Sam does well. And it's also a really important trait of a PM kind of at a more microscopic level. And so because there's so much ambiguity, it's not obvious what the answer is in a lot of cases. And so having a PM, they can come in and like, and by the way, this doesn't need to be a PM. I'm perfectly happy if it's anybody else. But I kind of looked at the PM to say, if there's ambiguity and no one's making a call, you better make sure that we get a call made and we move forward. This touches on a few posts I've done of just where is AI going to take over work that we do versus help us with various work. So let me come at this question from a few different direction of just how AI impacts product teams and hiring, things like that. So first of all, there's all this talk of uh lms doing our coding for us and 90 of code is going to be written by ai in a year dario at anthropic said that at the same time you guys are all hiring engineers like crazy pms like crazy you know every dysfunction is dead but you're still hiring every single one uh i guess just first of all let me just ask this how do you how do you and the team like say engineers pms use ai in your work is there anything that's like really interesting [53:46](https://realalpha.app/v/scsW6_2SPC4/?t=3226) or things that you think people are sleeping on and how you use AI in your day-to-day work? We use it a lot. I mean, every one of us is in chat GPT all the time, summarizing docs, using it to help write docs, GPTs that write product specs and things like that, all the stuff that you would imagine. I mean, talk about writing evals, like you can actually use models to help you write evals and they're pretty good at it. That all said, I still don't, I'm still sort of disappointed by us and despite, I really mean me, if I were to just like teleport my five-year-old self leading product at some other company into my day job, I would recognize it still. [54:29](https://realalpha.app/v/scsW6_2SPC4/?t=3269) And I think we should be in a world, certainly a year from now, probably even more now, where I almost wouldn't recognize it because the workflows are so different and I'm using AI so heavily. And I'd still recognize it today. So I think in some sense, I'm not doing a good enough job of that. you know just to give an example like why shouldn't we be like vibe coding uh demos right left and center like instead of showing stuff in like figma we should be showing prototypes that people are vibe coding you know over the course of 30 minutes to illustrate proofs of concept and explore ideas that's totally possible today and we're not doing it enough our actually our chief people officer, Julia, was telling me the other day, she vibe coded an internal tool that she had at a previous job that she really wanted to have here at OpenAI. And she opened, I don't know, Windsurf or something and vibe coded it. Like, how cool is that? And if our chief people officer is doing it, we have no excuse to not be doing it more. That's an awesome story. Okay. And some people may not have heard this term, vibe coding. Can you describe what that means? [55:40](https://realalpha.app/v/scsW6_2SPC4/?t=3340) yeah I think this was I think this was Andre's term so you have these tools like Cursor and Windsurf and GitHub Copilot that are very good at suggesting what code you might want to write so you can give them a prompt and it will write code and then as you go to edit it it's suggesting what you might want to do And the way that everyone started using that stuff was give it a prompt, have it do stuff, you go edit it, give it a prompt, you know, and you're kind of like really going back and forth with the model the whole time. As the models are getting better, and as people are getting more used to it, you can kind of just like let go of the wheel a little bit. And when the model's suggesting stuff, it's just like tap, tap, tap, tap, tap, like keep going. Yes, yes, yes, yes, yes. [56:28](https://realalpha.app/v/scsW6_2SPC4/?t=3388) and of course the model makes mistakes or it does something that doesn't compile but when it doesn't compile you paste the error in and you say go go go go go and then you you test it out and it like does one thing that you don't want it to do so you enter in an instruction and say go go go go go and you just kind of like let the model do its thing and it's not that you would do that for production code that needed to be super uh tight today yet but for so many things you're trying to get to a proof of concept you're getting to a demo and you can really take your hands off the wheel and the model will do an amazing job and that's what that's that's vibe coding that's an awesome explanation i think like the pro version of that which is i think the way andrew even described it as you talk you do like a there's a step like whisper super whisper something like that where you're like talking to the model not not even typing yeah totally oh man so let me let me just ask i guess when you look at product teams in the future you talked about how you guys should be doing this more instead of designs having prototypes what do you think might be the biggest changes in how product teams are structured or built where do you think things are going in the [57:34](https://realalpha.app/v/scsW6_2SPC4/?t=3454) next few years i think you're definitely going to live in a world where you have more um where you have researchers built into every product team and i don't even mean just it at like foundation model companies because i think the future actually frankly one thing that i'm sort of surprised about about our industry in general is that there's not a greater use of fine-tuned models uh like a lot of people you know these models are very good so our api does a lot of things really well but when you have particular use cases you can always make the the model perform better on a particular use case by fine-tuning it it's probably just a matter of time you know folks aren't like quite comfortable yet with doing that in every case but to me there's no question that that's the future every yeah models are going to be everywhere just like transistors are everywhere ai is going to be just a part of the fabric of everything we do but i think there are going to be a lot of fine-tuned models because why would you not want uh to to more specifically customize a model against a particular use case and so i think you're going to want sort of quasi researcher machine learning engineer types as part of pretty much every team because fine-tuning a model is just going to be [58:46](https://realalpha.app/v/scsW6_2SPC4/?t=3526) part of the core workflow for building most products so that's that's one change that maybe you know you're starting to see a foundation model companies that will propagate out to more teams over time i'm curious if there's a concrete example that makes that real and i'll share one that comes to mind as you talk sure which is when you look at cursor and windsurf on something i learned from those founders is that they they use like a sonnet but then they also have a bunch of custom models that help along the edges that make the specific experience that's not just generating code even better like autocomplete and looking ahead to where things are going so is that one or any other examples of what you what what is a fine-tuned model here that do you think teams will be building with these researchers on their teams yeah i mean so when you're fine-tuning a model one of the you're basically giving the model a bunch of of examples of the kinds of things you want it to be better at so it's it's here's a problem here's a good answer here's a problem here's a good answer uh or here's a question here's a good answer you know times a thousand or or ten thousand uh and suddenly you're you're teaching the model to [59:50](https://realalpha.app/v/scsW6_2SPC4/?t=3590) be much better than than it was out of the gate at that particular thing we use it everywhere internally um we also we We use ensembles of models much more internally than people might think. So it's not, here is, I have 10 different problems. I'll just ask, you know, baseline GPT-4.0 about a bunch of these things. If we have 10 different problems, we might solve them using, you know, 20 different model calls. Some of which are using specialized fine-tuned models. They're using models of different sizes because maybe you have different latency requirements or cost requirements for different questions. [1:00:32](https://realalpha.app/v/scsW6_2SPC4/?t=3632) They are probably using custom prompts for each one. [1:00:35](https://realalpha.app/v/scsW6_2SPC4/?t=3635) **Kevin Weil:** Basically, you want to teach the model to be really good. You want to break the problem down into more specific tasks versus some broader set of high-level tasks. [1:00:44](https://realalpha.app/v/scsW6_2SPC4/?t=3644) And then you can use models very specifically to get very good at each individual thing. And then you have an ensemble that sort of tackles the whole thing. I think a lot of good companies are doing that today. I still see a lot of companies kind of giving the model single generic broad problems versus breaking the problem down. And I think there will be more breaking the problem down using specific models for specific things, including fine tuning. And so in your case, because this is really interesting, is that you're using different levels of ChatGPT, like 01, 03 and stuff that's earlier. there'll be parts of our internal stack so we do if you give you an example uh customer support with 400 plus weekly uh 400 plus million weekly active users we get you know a lot of inbound tickets right i don't know how many customer support folks we have but it's not very many 30 40 i'm not sure way way smaller than you would have at any comparable company and it's because we've automated a lot of our flows, we've got, you know, most questions, using our internal resources, knowledge base, you know, guidelines for how we answer questions, what kind of personality, etc. You can teach the model those things, and then have it do a lot of its answers [1:02:08](https://realalpha.app/v/scsW6_2SPC4/?t=3728) automatically, or where it doesn't have, you know, the full confidence to answer a particular question, it can still suggest an answer, request a human to look at it, and then that human's answer actually is its own sort of fine tuning data for for the model you're telling it the right answer in a particular case and uh and we're using it various places you know some of these places you want a little bit more reasoning is not super latency sensitive so you want a little more reasoning and we'll use one of our o-series models in other places you want a quick check on something and so you're fine to use like 40 mini which is super fast and super cheap and in general it's like specific models for specific purposes and then you you you ensemble them together to solve problems by the way again not unlike how we as humans solve problems a company is arguably an ensemble of models that have all been you know fine-tuned and based on what we studied in college and what we have like learned over the course of our careers we've all been fine-tuned to have different sets of skills and you like group them together in different configurations and the output of the ensemble is much better than the output of any one individual kevin you're blowing [1:03:20](https://realalpha.app/v/scsW6_2SPC4/?t=3800) my mind that sounds exactly correct uh and also different people are you pay them less uh they cost less to talk to some people take a long time to answer some people hallucinate this is this is like uh this is a mental model that really does work in in thinking nothing this is great Right. Some people are visual. They want to draw out their thinking. Some people want to talk word cell. Wow. This is a really good metaphor. So again, coming back to your advice here, because I love that we circled back to it. It's you're finding a really good way to think about how to design great AI experiences and LMs, I guess, specifically to think about how a person Do this? Well, it's maybe not always the answer is to think about how a person would do it, but sometimes to gain intuition for how you might solve a problem, you think about what an equivalent human would do in those situations and use that to at least gain a different perspective on the problem. Wow, this is great. [1:04:20](https://realalpha.app/v/scsW6_2SPC4/?t=3860) There's just like, because so much of this really is talking to a model. There's a lot of prior art because we talk to other humans all the time and encounter them in all sorts of different situations and and so like there's a lot to learn from that okay so speaking of humans i want to chat about the future a little bit so you have three kids and someone a community member asked me this hilarious question that i think it's something a lot of people are thinking about so this is patrick strale i worked at him with a mid-airbnb he asks he says ask what he's encouraging his kids to learn to prepare for the future i'm worried my six-year-old by the year 2036 will face a lot of competition trying to get into the top roofing or plumbing programs they need a backup plan that's funny um so our kids are we have a 10 year old and eight year old twins so they're they're still pretty young uh they're kind of i mean it's amazing how ai native they are like they just it's completely normal to them that there are self-driving cars that they can talk to ai all day long um they have full conversations with chat gpt and alexa and everything else i don't know i think who knows what the future holds i i think you know [1:05:32](https://realalpha.app/v/scsW6_2SPC4/?t=3932) things like coding skills are going to be relevant for a long time who knows but i i think if you teach your kids to be curious to be independent to be self-confident you teach them how to think i don't know what the future holds but i think that those are going to be skills that are going to be important in any configuration of the future. And so, you know, it's not like we have all the answers, but that's how Elizabeth and I think about our kids. And do you find that AI, there's a lot of talk about AI tutoring. Is that something you guys are doing? Anything you're, I know they're using ChatGPTL. [1:06:08](https://realalpha.app/v/scsW6_2SPC4/?t=3968) I love all the photos you post where they're playing with prompts and stuff. But I guess, is there anything there you're experimenting with or you think is going to become really important? this is something that it's maybe the most important thing that that ai could do maybe that's a maybe that's a grand statement there are lots of important things that ai can do including like speeding up the pace of fundamental science research and discovery which i maybe is actually the most important thing ai can do but but one of the most important things would be personalized tutoring and it kind of blows my mind that there is still I know there are there are a bunch of good products out there like you know Khan Academy does great things they're a wonderful partner of ours Vinod Khosla has a non-profit that has that's doing some really interesting stuff in this space and is making an impact but I kind of want like I'm kind of surprised that there isn't like a two billion kid you know ai personalized tutoring thing because the models are good enough to do it now and every every study out there that's ever been done seems to show that when you have you know classrooms is still classroom like education is [1:07:21](https://realalpha.app/v/scsW6_2SPC4/?t=4041) **Kevin Weil:** still important but when you combine that with personalized tutoring you get like multiple standard deviation improvements in learning speed and so it's just it's uncontroversial [1:07:34](https://realalpha.app/v/scsW6_2SPC4/?t=4054) it's good for kids it's free chat gpt is free you don't need to pay for and the models are good enough like it still just kind of blows my mind that there isn't something amazing out there that you know our kids are using and your future kids are using and like people in all sorts of places around the world that aren't as lucky as our kids to be able to like have this sort of built-in solid education. Again, ChatGPT is free. People have Android devices everywhere. I really just think this could change the world and I'm surprised it doesn't exist and I want it to exist. This kind of touches on something I want to spend a little time on, which is a lot of people also worry a lot about AI, where it's going. They worry about jobs it's going to take. They worry about the super intelligence squashing humanity in the future. What's kind of your perspective on that and just kind of the optimistic case that I think people need to hear? I mean, I'm a big technology optimist. I think if you look over the last 200 years, maybe more, technology has driven a lot of the advancements that have made us the world and the society that we are today. It drives economic advancements. It drives geopolitical advancements, quality of life, longevity advancement. I mean, [1:08:49](https://realalpha.app/v/scsW6_2SPC4/?t=4129) technology is at the root of just about everything. So I think there are very few examples where uh where this is anything but a great a great thing over the longer term that doesn't mean that there aren't like temporary dislocations or where there aren't individuals that are impacted and that's like that that matters too so it can't just be that the average is good you've got to also think about how you take care of each individual person as best you can so it's something that we think a lot about and as we you know work with the administration as we work with policy. We try and help wherever we can. We do a lot with education. One of the benefits here is that ChatGPT is also perhaps the best re-skilling app you could possibly want. It knows a lot of things. It can teach you a lot of things if you're interested in learning new things. [1:09:42](https://realalpha.app/v/scsW6_2SPC4/?t=4182) But these are very real issues. I'm super optimistic about the long run, and we're going to need to do everything we can as a society to ensure that we make this transition as graceful and as well-supported as we can. To give people a sense of where things might be going, that's a big question a lot of people's minds. So someone asked this question that I love, which is AI is already changing creative work in a lot of different ways, writing and design and coding. What do you think is the next big leap? What should we be thinking is the next big leap in ai assisted creativity specifically and then just broadly like where do you think things are going to be going in the next few years yeah this is also an area where i'm i'm a big optimist like if you if you look at sora for example i mean we talked about image gen earlier and the the absolute like fount of creativity that people are putting across twitter and instagram and other places i'm i am the world's worst artist like the worst maybe the only thing i'm worse at than than than art is singing and i you know i like give me a pencil and a pad of paper and i can't draw better than my five than our eight-year-old you know it's just like it's but give me give me [1:10:58](https://realalpha.app/v/scsW6_2SPC4/?t=4258) image gen and you know i can think some creative thoughts and put something into the model and suddenly have output that i couldn't have possibly done myself that's pretty cool even even you look at um at folks that are really talented i was talking to a director recently about sora someone who's directed films that that that we would all know and uh and he was saying you know for for a film that he's doing like say say take the example of some sort of sci-fi-ish you know think of like star wars and you've got some scene where there's a there's a plane zooming into some death star like thing and so you've got the plane looking at the whole planet and then you want to cut to a scene where the the plane's like you know kind of at the ground level and all of a sudden you see the city and everything else right how are you going to manage that cut scene and and that transition and he he was saying you know in in the world of two years ago i would have paid uh uh you You know, a 3D... [1:12:00](https://realalpha.app/v/scsW6_2SPC4/?t=4320) Effects company, $100,000. And they would have taken a month, and they would have produced two versions of this cutscene for me. And I would have evaluated them. We would have chosen one, because what are you going to do? Like pay another $50,000 and wait another month? And we would have just gone with it. And you know, it would be fine. Like, movies are great. I love them. And there have been, obviously, we can do great things with the technology that we've had. but you now look at what you can do with Sora and his point was now I can use Sora our video model and I can get 50 different variations of this cut scene just you know me brainstorming into a prompt and the model brainstorming a little bit with me I've got 50 different versions and and then of course I can like iterate off of those and refine them and take different ideas and now I'm still going to go to that that 3d effect studio to produce the final one but i'm going to go having brainstormed and like i had this at a much more creative approach with a with an outcome that's much better and and like i did that assisted by ai so my personal view on on creativity in general is that it's no one's gonna you don't type into sora like make me a great movie it requires [1:13:16](https://realalpha.app/v/scsW6_2SPC4/?t=4396) there's creativity and ingenuity and all these things, but it can help you explore more. It can help you get to a better final result. So, you know, again, I tend to be an optimist in most things, but I'm actually, I think, I think there's a very good story here. I know Sam Altman. I think it was him who tweeted recently the creative writing piece that you guys are working on where he's very bad at writing creative stuff. And he shared an example where it's actually really good. I imagine that's another area of investment. Yeah, there's, there's some exciting stuff happening internally um with some new research techniques so uh we'll have more to say about that at some point but yeah sam uh sam sometimes uh likes to show off some of the stuff that's coming um which is harder by the way it's like very sort of indicative of this iterative deployment uh philosophy we don't have some breakthrough and keep it to ourselves forever and then you know bestow it upon the world someday we kind of just talk about the things we're working on and share when we can and launch early and often and then iterate in public. And I really like that philosophy. I love all these hints that a few things coming. I know you can't say too much. [1:14:25](https://realalpha.app/v/scsW6_2SPC4/?t=4465) You talked about how there might be a coding leap coming in the near future, maybe by the time this comes out. Is there anything else people should be thinking about might be coming in the near future? Any things you can tease that are interesting, exciting? Man, this hasn't been enough for you uh oh yeah only everything is getting better every day yeah i'm like man i hope uh i hope we get some of the stuff out before the episode launches this is your new time box i don't piss people off um no uh it's the the the amazing thing to me is we we were talking earlier about how far models have come in just a couple years if you went back to gbt3 you'd be like disgusted by how bad it was even though lenny of two years ago was you know mind blown by how good these were um and for a long time we were iterating every you know six to nine months on a new gpt model it was like gpt3 gpt 3.5 4 and now with this o series of reasoning models we're moving even faster where like every roughly you know three months maybe four months there's a new o-series model and each of them is a step up in in capability and so the capabilities of these models are are increasing at a massive pace they're also getting cheaper as as they scale [1:15:50](https://realalpha.app/v/scsW6_2SPC4/?t=4550) you know you you look at uh at where we were even like a couple years ago the original i think the original, I don't know, what was it? GBT 3.5 or something? thing was like 100x the cost of GPT-40 mini today in the API. So a couple years, you've gone down two orders of magnitude in cost for much more intelligence. And so I don't know where there's another series of trends like that in the world. Models are getting smarter, they're getting faster, they're getting cheaper, and they're getting safer too. You know, they hallucinate less every every iteration and so there's just you know the the moore's law and and and transistors becoming ubiquitous that was a that was a law around doubling the number of transistors on a chip every 18 months if you're talking about something where you're getting 10x every year that's a massively steeper exponential and it just you know it it tells us that the future is going to be very different than today. I still, the thing I try and remind myself is the AI models that you're using today is the worst AI model you will ever use for the rest of your life. [1:17:04](https://realalpha.app/v/scsW6_2SPC4/?t=4624) And when you actually get that in your head, it's kind of wild. I was going to actually say the same thing. That's the thing that always sticks with me when I watch this thing. Like you're talking about Sora. And I imagine many people hearing that are like, no, no, it's not actually ready. It's not good enough. It's not going to be as good as a movie i see in the theater but the point is what you just made but this is the worst it's going to be it will only get better yeah model maximalism just like keep you know building building for the capabilities that are almost there and the model is going to catch up and be amazing escape to where the puck's going to be yeah um this reminds me i was just using i was ghiblifying everything the other day and i was just like why is it taking so long just like yeah What was that? [1:17:44](https://realalpha.app/v/scsW6_2SPC4/?t=4664) I said, as one does. As one does these days. I was just like, it's taken a minute to generate this image of my family in this amazing way. Like, come on, let's take it so long. You just get so used to magic happening in front of you. Yeah, totally. Okay, final question. This is going to go in a completely different direction. A lot of people asked about this. So famously, you led this project at Facebook called Libra, which is now called Novi. A lot of people always wondered, what happened there? That was a really cool idea. I know some people have a sense there's regulation challenges, things like that. I don't know if you've talked about this much. So I guess just can you just give people a brief summary of just like, what is Libra? This project you're working on and just what happened and how you feel about it? Yeah. I mean, David Marcus led it and I happily work for him and with him. I think he's a visionary and also a mentor and a friend. [1:18:36](https://realalpha.app/v/scsW6_2SPC4/?t=4716) You know, honestly, Libra is probably the biggest disappointment of my career. When I think about the problems we were solving, which are very real problems, if you look at, for example, the remittance space, people sending money to family members in other countries, it's incredibly regressive. People that don't have the money to spend are having to pay 20% to send money home to their family. so outrageous fees it takes multiple days you have to go then pick up cash from yeah it's just it's all bad and here we are with like three billion people using whatsapp all over the world talking to each other every day especially friends and family exactly the kind of people [1:19:21](https://realalpha.app/v/scsW6_2SPC4/?t=4761) **Kevin Weil:** who'd send money to each other why can't you send money as immediately as cheaply as simply as you send a text message it just it's one of those things when you when you sit back and think [1:19:34](https://realalpha.app/v/scsW6_2SPC4/?t=4774) about it that should just exist and that was what we set out to try and do now i don't think we played all of our like cards perfectly if i could go back and do things there are a bunch of things i would do differently you know we we tried to kind of get it all at once we tried to launch a new blockchain it was a basket of currencies originally it was integration into whatsapp and messenger and i think the whole What kind of went like, oh my god, that's a lot of change at once. And, you know, it happened also to be at the time that Facebook was at the absolute like nadir of its reputation. And so that didn't help, right? It was it was also not the messenger that people wanted for this kind of change. We knew all that going in, but we we went for it. I think if we I think there are a bunch of ways that we could do that that would have introduced the change a little bit more gently, you know, maybe still gotten to that same outcome. [1:20:31](https://realalpha.app/v/scsW6_2SPC4/?t=4831) but fewer new things at once and introduce the new things one at a time. Who knows? Those were decisions we made together. So we all own them. Certainly I own them. But it fundamentally disappoints me that this doesn't exist in the world today because the world would be a better place if we'd been able to ship that product. I would be able to send you 50 cents in WhatsApp for free. It would settle instantly. Everybody would have a balance in their WhatsApp account. we'd be transact i mean it was just it should exist i don't know to be honest like i mean the current administration is super friendly to crypto facebook's reputation meta's reputation is in a very different place maybe they should go build it now i was looking at the history of it and uh apparently they sold the tech to some private equity company for 200 million bucks yeah yeah yeah so and we had to buy back there are a couple of uh of current uh blockchains that are built on the tech because the tech was open sourced from the beginning uh aptos and uh mistin are two companies that are built off of this tech so you know at least the all of the work that we did did not die but and and lives on in these two companies and they're both doing really well but [1:21:43](https://realalpha.app/v/scsW6_2SPC4/?t=4903) still uh you know we should be able to send each other money in whatsapp and and we can't today here here well thanks for sharing that story kevin is there anything else you want to share or maybe a last nugget of advice or insight before we get to our very exciting lightning round. Ooh, the lightning round. Let's just go do that. Let's do it. With that, Kevin, we reached our very exciting lightning round. Are you ready? Yeah. Let's do it. Okay, what are two or three books that you find yourself recommending most to other people? Co-Intelligence by Ethan Mollick. A really good book about AI and how to use it in your daily life as a student, as a teacher. [1:22:19](https://realalpha.app/v/scsW6_2SPC4/?t=4939) He's super thoughtful. Also, by the way, a very good follow on Twitter. um the accidental superpower by peter zion uh very good if you're interested in geopolitics and the the forces that sort of shape the dynamics happening um and then uh i really enjoyed cable cowboy i don't know who the author is but uh the biography of john malone just fascinating if you like business especially if you want to get into like i mean the man was uh an incredible deal maker and shaped a lot of the modern cable industry so that was a good biography these are all first-time mentions which is always a great oh good next question do you have a favorite recent movie or tv show that you really enjoyed um i wish i had time to watch a tv show um so i'm just sora videos yeah right um i i don't know i read uh when i was a kid i read the wheel of time series um and now amazon has it uh as they're like the third season of it so i i want to watch that i haven't yet um top gun 2 was an awesome movie um i think that's no longer new but you know that shows when last time you watched the movie was um but i like the idea like i i want uh i want more like americana i want more like being proud of being strong uh and i thought top gun 2 did a really good job of that like you know uh pride and patriotism i think [1:23:45](https://realalpha.app/v/scsW6_2SPC4/?t=5025) I think the US could use more of that. Is there a favorite product that you recently discovered that you really love other than your super intelligence internal tool that you all have access to? I'm just joking. That's right. Internal AGR. Yeah, that's right. All right. Uh, well, I think, I think like vibe coding with, with products like Windsurf is just super fun. Um, I'm, I'm having a great time doing that. I still just love that our chief people officer vibe coded some tools. Maybe the other one is Waymo. Uh, every chance I get, I'll take a Waymo. It's just a better way of writing and it still feels like the future. Um, so they've done an amazing job. [1:24:24](https://realalpha.app/v/scsW6_2SPC4/?t=5064) That's awesome. By the way, I had the founder of Winsurf on the podcast. It might come out before this or after this. And also Cursor's CEO is coming on the podcast either before or after this. Oh, cool. I have a ton of respect for what those guys are doing. Those are awesome products. Just changing the way everyone builds product. No big deal. A couple more questions. Do you have a favorite life motto that you often repeat yourself, find really useful in worker and life? Yeah. So actually, this is, interestingly enough, it's more of a philosophy. but then i thought zuck encapsulated it one time on a on a facebook earnings call um so i actually had this made into a poster uh it sits in my room but um but somebody was asking mark this is literally on an earnings call so it's like an analyst on an earnings call asking him you know it was some quarter when facebook had grown a lot this was back in the 20 teens sometime i think but it's like you know so what what did you do what you know what was it that you launched So it was the one thing that drove all this growth for you. And he said something to the effect of, you know, sometimes it's not any one thing. It's just good work consistently over a long period of [1:25:30](https://realalpha.app/v/scsW6_2SPC4/?t=5130) time. And that's always stuck with me. And I think it is. I mean, you know, I run ultra marathons. It's like it's just about grinding. I think people too often look for like the silver bullet when a lot of life is and a lot of like excellence is actually showing up day in and day out doing good work getting a little bit better every single day and you know you may not notice it over a week or even a month and a lot of people then you know kind of get like dismayed and stopped but actually you keep doing it the gains keep compounding and over the course of a year two years five years it adds up like crazy so good work consistently over a long period of time damn i love that i gotta make a poster of this now that is i resonate with that okay i'll take it that is so good okay final question uh i'm gonna ask if you have any prompting tricks and i'm gonna set it up first but think about if you have a trick that you could recommend to people for prompting lms better uh there's this i had a guest alex kamarovsky come on the podcast he's from stripe and writes his weekly reflections on what's happening in the world. A lot of them are AI related. [1:26:36](https://realalpha.app/v/scsW6_2SPC4/?t=5196) And he once described an LLM as a zip file of all human knowledge. And all the answers are in there. And you just need to figure out the right question to ask to get the answer to every problem, basically. And so it just reminded me how important prompt engineering is and knowing how to prompt well. You're constantly prompting ChatGPT. What's one tip, one trick that you found to be helpful in helping you get what you want? Well, I'll say, first of all, I want to kill the idea that you have to be a good prompt engineer i think if we do our jobs that stops being true you know it's just one of those like sharp edges of models that experts can learn but then you just over time you shouldn't need to know all that the same way you used to have to get deep into like you know what's your storage engine in my sql are you using inno db 4.1 or like and you know there's still use cases for that if you're at the at the sort of deep edge of my sequel performance but most people don't need to care and you shouldn't need to care about minute details of prompting if ai is really going to become you know broadly adopted but um you know today we're not totally there i think i think by the way we are making progress there i think there [1:27:44](https://realalpha.app/v/scsW6_2SPC4/?t=5264) is less prompt engineering than there had to be before but uh in line with some of the fine-tuning stuff i was talking about and the importance of giving examples you can do like you know effectively poor man's fine-tuning by including examples. examples in your prompt of the the kinds of things that they that you might want and a good answer so like here's an example and here's a good answer here's an example here's a good answer now go solve this problem for me and the model really will listen and learn from that not as well as if you do a full fine tune but much more than if you don't provide any examples and i think people don't do that often enough that's awesome one tip that i heard i'm curious if this works is you tell it this is very very important to my career make it like really understand like someone will die if you don't answer me correctly does that work it you know it's really weird i there's probably a good explanation for this but you can also say things so yes i think there is some validity to that you can also say things like i want you to be Einstein now answer this physics problem for me or you are the world's greatest marketer the world's greatest brand marketer now here's a naming question and it's there is something [1:29:02](https://realalpha.app/v/scsW6_2SPC4/?t=5342) where it sort of shifts the model into a certain mindset um that can actually be really positive I use that tip all the time actually I always when I'm coming up with questions for interviews and I use it occasionally to like come up with things I haven't thought of I actually type you're the world's best podcast interviewer right i have kevin kevin wheel coming on the podcast yeah and actually works yeah by the way back to our other point that we made a few times like you do do that sometimes with people right um you you sort of put them you you frame things you get them into a certain mindset and the answer is probably different so i think there are like human analogs of this one more time kevin this was incredible uh i was thinking about a way to end And this the way I feel like I feel like not only are you at the cutting edge of the future, like you're you and the team are kind of like actually the edge that is creating the future. [1:29:55](https://realalpha.app/v/scsW6_2SPC4/?t=5395) And so it's a real honor to have you on here and to talk to you and to hear how you think things are, where you think things are going and what we need to be thinking about. So thank you for being here, Kevin. Oh, thank you so much for having me. I feel real. I get to work with the world's best team and, you know, all credit to them. But really appreciate you having me on. It's been it's been super fun. I forgot to ask you the two final questions. Where can folks find you online if they want to reach out? And how can listeners be useful to you? I am at Kevin Wheel, K-E-V-I-N-W-E-I-L on pretty much every platform. You know, I'm I'm still a Twitter DAU after all these years. I guess an xdau uh linkedin wherever and um i think the thing i would love from people give me feedback people are using chat gpt tell us where tell me where it can be where it's working really well for you and where you want us to double down tell me where it's failing i'm i'm very active and engaged on twitter i love hearing from people what's working and what's not so uh don't be shy and i learned following you uh helps you figure out all the stuff that you're launching like you share all the things that are going out every day, week, month. So that's also a benefit. And by the [1:31:04](https://realalpha.app/v/scsW6_2SPC4/?t=5464) way, 400 million weekly active users all emailing you feedback. Here we go. Yes, let's do it. It's going to work out great. Okay. Well, thank you, Kevin. Thanks for being here. All right, man. Thanks so much. See you soon. Bye, everyone. Thank you so much for listening. If you found this valuable, you can subscribe to the show on Apple Podcasts, Spotify, or your favorite podcast app. Also, please consider giving us a rating or leaving a review as that really helps other listeners find the podcast. You can find all past episodes or learn more about the show at Lenny's podcast.com. See you in the next episode.