Everyday AI still depends on access to real-world systems

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 40:51

   While we stay on benchmarks, another super cool benchmark, this one out from OpenAI launched this week. It's called LifeVal. And to be clear, this is OpenAI's own benchmark that they're measuring against their own model. So while GDPX asked whether AI can do an expert's job, and this benchmark asked the question, can it help you? Can OpenAI's models help you with everyday tasks? So OpenAI surveyed 3,995 Americans about their everyday tasks and asked them where they want help.

2. 41:27

   They built 169 real-world tasks around those answers. So finding childcare, enrolling a kid in kindergarten, creating a household budget, ordering groceries, dealing with government agencies, modifying a recipe, designing a workout plan. The AI has to do the task using a browser and a computer like a human assistant would. So let me share the chart here, and then let's talk about it because, you know, this is where AI starts touching everybody's real life.

3. 41:59

   So here we are. In December of 2025, GPT 5.2 scored 49.3. In September of 2026, last month, GPT 6 Astra scored 80.6. That's a jump from about half of the errands being completed to 80 percent in nine months. So what does this mean? You know, I think this is a everybody's getting an executive assistant that knows their needs. Again, elephant in the room here. It's opening. AI's own benchmark scored against their own models.

4. 42:34

   GDP access, we mentioned, is independent. So, Salim, you know, which matters more in this world? AI's that do 80% of everybody's errands or the 12% of expert tasks? I mean, sort of, you know, I'm asking that. You know, I thought for me this was the most exciting thing on our docket today because it's one thing, not everybody needs a superhuman AI to solve complex mathematics, right? But almost everybody can benefit from a competent personal assistant.

5. 43:04

   Imagine you're a single mother trying to navigate kids' healthcare plans, government, bureaucracy, and you're losing your mind. Yeah, and so having that, it's like a way of getting to abundance. You get to abundance in a totally different way by allowing people, giving people help in the areas that need most. And I thought this was a really great way for OpenAI to come down the stack a bit and help people out.

6. 43:29

   Because, you know, the GDPR thing is measuring like some of the hardest jobs in the economy. This is measuring, can you, the friction of being human, right? And we all have that, everybody has that and democratizing that, giving clear access to that will really, really make a very big difference. Very exciting. You know, we get, you know, probably 90% positive comments on, in our YouTube videos, But, you know, there's a small percentage of folks who basically say we're disconnected.

7. 43:57

   You know, we're all billionaires. We're not all billionaires, to be very clear. Maybe you are, Dave. I don't know. But only some of us are billionaires. OK. But, you know, this is where the rubber hits the roads for, you know, 90 percent of Americans having this kind of capability effectively for free. And people need to start utilizing it that way. Alex, you want to talk about this chart that's up?

8. 44:23

   Yeah, this is really striking as a chart. So, first feature I want to call to everyone's attention, look at the enormous gap on the frontier. So, this is, for those who can't see, this is a price versus performance, although the price is measured in latency. Price versus performance chart for multiple GDP models for this lifestyle, I guess, evaluation. Total contrast to GDPX. GDPX is based on hard frontier stem tasks this seems to be based on quote-unquote everyday tasks look at that enormous gap between gpt 5.2 and then the cluster a pretty tight cluster between everything from GPT 5.4 to GPT 6 Astra. That tells me a few things.

9. 45:08

   One, or rather, I should say, it points in a few different suggestive directions. So possibility A, OpenAI maybe just gave up on optimizing GPT models for whatever they construe as everyday tasks after GPT 5.4. Certainly, there's an easy story that one could tell there. Maybe this is in connection with OpenAI's notorious pivot from the consumer back to enterprise. And maybe OpenAI just doesn't care or didn't care after GPT 5.4 as much about optimizing for consumer CUA use cases.

10. 45:42

    That's possibility A. Possibility B is that maybe everyday tasks, whatever that means, are just really hard. Maybe the first 80% of them are relatively easy, and the remaining 20% are really hard. However, if that were the case, I would still expect to see a bigger spread between GPT 5.4 and GPT 6 Astra. So what I come away from this chart scratching my head over is maybe this is unintentionally by OpenAI's part in releasing this benchmark.

11. 46:15

    Maybe they're basically signaling to the market, this is not, even though we're releasing a benchmark on it, we don't actually care as much about this benchmark as we do about enterprise benchmarks. Well, I'll give you a case study that came up on one of the analysts yesterday on the EverQuote earnings call was saying, look, I spent four hours trying to just relocate my insurance policies on the phone with customer service agents. Are we not living in the age of AI? How could I possibly have wasted four hours yesterday on this? And the answer is the AI, you know, GPT 5.2 couldn't do it, but now the ai is more than capable of doing it but the carriers are not connected to the ai and they don't they're just confused still they don't know how to interact with this metamuse is imminent and people are going to want to go to metamuse to get everything done

12. 47:01

    including buy auto insurance or whatever and if it's not connected to the back-end systems it can it can walk you through it and then it leaves you talking to a human being and then you lose the four hours again so that that connection point is still broken in society and it's all hung up in people's fears and their confusion and their inability but you know and very specifically if i make it really easy for through metamuse to get everything you want in life how am i making money Facebook. Like, how am I getting paid in the middle of that transaction? When it's a paid click, it's really obvious how that economy evolved. Right now, the AI version of it is very much in flux, and that's holding it back. So a lot of the AI capabilities are getting way, way, way ahead of their connection to the world. And that's true. That's true in customer service.

13. 47:49

    It's true in robotics. It's true. And that's one of the reasons Kevin Weil was talking to him this week about his new fully automated biotech research company. It got massively funded