# Compute-first learning is overtaking human-engineered AI

Markdown copy of https://realalpha.app/s/compute-first-learning-is-overtaking-human-engineered-ai/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Chamath Palihapitiya](https://realalpha.app/people/chamath-palihapitiya/) (Host) · Founder & CEO, Social Capital · Led Facebook's growth team; his firm backed Slack and AI chipmaker Groq.
- Host: [Jason Calacanis](https://realalpha.app/people/jason-calacanis/) · Founder & CEO, LAUNCH · Early Uber angel; sold Weblogs Inc. to AOL; hosts This Week in Startups.
- Host: [David Friedberg](https://realalpha.app/people/david-friedberg/) · CEO, Ohalo · Founded The Climate Corporation, sold to Monsanto for about $1.1B in 2013.
- Show: All-In
- Episode: [Grok 4 Wows, The Bitter Lesson, Third Party, AI Browsers, SCOTUS backs POTUS on RIFs](https://realalpha.app/v/KypnjJSKi4o/), Jul 11, 2025
- Moment: 21:03 to 29:23 of the episode (8 min), playing on https://realalpha.app/s/compute-first-learning-is-overtaking-human-engineered-ai/; the episode from there: https://realalpha.app/v/KypnjJSKi4o/?t=1263

## Quote

> “you're always better off when you're trying to solve an AI problem taking a general learning approach that can **scale with computation**”
> — [Chamath Palihapitiya](https://realalpha.app/people/chamath-palihapitiya/) at [21:37](https://realalpha.app/v/KypnjJSKi4o/?t=1297) · All-In

## Takeaways

1. General learning can outperform hand-coded human expertise.
2. More compute and data helped xAI catch up quickly.
3. Human-generated training data may have a short half-life.

## About this segment

Chamath Palihapitiya explains the Bitter Lesson: general learning scaled with computation can beat systems built around human knowledge. The debate explores what that means for AI training and self-driving.

From “Grok 4 Wows, The Bitter Lesson, Third Party, AI Browsers, SCOTUS backs POTUS on RIFs”, Jul 11, 2025. [Watch the full episode](https://realalpha.app/v/KypnjJSKi4o/)

## Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[21:10](https://realalpha.app/v/KypnjJSKi4o/?t=1270) But the second is a fundamental architectural decision that Elon made, which I think we didn't fully appreciate until now. And it maps to an architectural decision he made at Tesla as well. And for all we know, we'll figure out that he made an equivalent decision at SpaceX. And that decision is really well encapsulated by this essay, The Bitter Lesson, by Rich Sutton. And Nick, you can just throw this up here.

[21:39](https://realalpha.app/v/KypnjJSKi4o/?t=1299) But just to summarize what this says, it basically says in a nutshell that you're always better off when you're trying to solve an AI problem taking a general learning approach that can scale with computation because it ultimately proves to be the most effective. And the alternative would be something that's much more human labored and human involved that requires human knowledge. And so the first method, what it essentially allows you to do is view any problem as an endless scalable search or learning task.

[22:16](https://realalpha.app/v/KypnjJSKi4o/?t=1336) And as it's turned out, whether it's chess or Go or speech recognition or computer vision, whenever there was two competing approaches, one that used general computation and one that used human knowledge, the general computation problem always won. And so it creates this bitter lesson for humans that want to think that we are at the center of all of this critical learning and all of these leaps. In more AI-specific language, what it means is that a lot of these systems create these embeddings that are just not understandable by humans at all, but it yields incredible results. So why is this crazy? Well, he made this huge bet on this 100,000 GPU cluster, people thought, wow, that's a lot, is it going to bear fruit? Then he said, No, actually, I'm scaling it up to 250,000. Then he said, it's going to scale up to a million. And what these

[23:09](https://realalpha.app/v/KypnjJSKi4o/?t=1389) results show is a general computational approach that doesn't require as much human labeling, can actually get to the answer and better answers faster. That has huge implications. Because if you think about all these other companies, what has Lama been doing? They just spent $15 billion to buy 49% of scale AI. That's exactly a bet on human knowledge. What is Gemini doing? What is OpenAI doing? What is Anthropic doing? So all these things come into question. And then the last thing I'll say is, if you look back, he made this bet once before, which was Tesla FSD versus Waymo. And Tesla FSD only had cameras. It didn't have LiDAR. But the bet was, I'll just collect billions and billions of driving miles before anybody else does.

[24:00](https://realalpha.app/v/KypnjJSKi4o/?t=1440) and apply general compute and it'll get to autonomy faster than the other more laborious and very expensive approach so i just think it's an incredible moment in technology where we see so many examples travis is another one what he's just talked about you know the bitter lesson is you could believe that you know food is this immutable thing that's made meticulously by hand by these individuals or you can take this general purpose compute approach which is what took, waited for these constructors to come into play. And now you can scale food to every human on earth. I just think it's a, it's so profoundly important. One thing I'll throw out there, Chamath, is the Tesla approach for autonomy is taking human knowledge. In fact, the whole idea is to approximate human, human driving, right, is the whole damn thing. Now, depending on your

[24:55](https://realalpha.app/v/KypnjJSKi4o/?t=1495) approach in the technology can do like what's called an end to end approach, or you can look at, okay, perception, prediction, planning, and control, which are like these four modules that sort of you, you, you sort of engineer, if that makes sense. But it's approximating human driving to do it. The difference is that, you know, I think Elon's taken a almost a more human approach, which is like I've got two eyes why can't my car why can't my car do it like a human like I don't have any lidar spinning around on my head as a human why can't my car so it's kind of interesting he's sort of taking what you're saying Chamath on the computation side because hardware five is coming out on Tesla probably next year which is going to make a big difference in what FSD can do that's the compute side you're talking about but then he is approximating human

[25:51](https://realalpha.app/v/KypnjJSKi4o/?t=1551) Yeah, I just meant that, you know, other than the first versions of FSD, which I think Andre talked about, Andrej Karpathy talked about, you know, they're not really so reliant anymore on human labeling, per se, right? So that's that interference. And then the other crazy thing that he said, subsequent versions of Grok are not going to be trained on any traditional data set that exists in the wild.

[26:17](https://realalpha.app/v/KypnjJSKi4o/?t=1577) The cumulative sum of human knowledge has been exhausted in AI training. That happened basically last year. And so the only way to then supplement that is with synthetic data where the AI creates, it'll sort of write an essay or it'll come up with a thesis and then it'll grade itself and sort of go through this process of self-learning with synthetic data. He said that he's going to have agents creating synthetic data from scratch that then drive all the training, which I just think is, it's crazy.

[26:48](https://realalpha.app/v/KypnjJSKi4o/?t=1608) Just explain this concept one more time with a bitter lesson. Hand coding heuristics into the computer and saying, hey, here's specific openings in chess. Yeah, use chess, right? Yeah, so you're hand coding specific examples of openings in there, end games, et cetera, versus just saying, play every possible game and here's every game we have. so here's yeah yeah so the two approaches would be let's say like travis and i were building competing versions of a chess solver and travis's approach would say i'm just going to define the chess board i'm going to give the players certain boundaries in which they can move right so the bishop can only move diagonally and there's a couple of boundary conditions and i'm going to create a reward function and i'm just going to let the thing self-learn and

[27:35](https://realalpha.app/v/KypnjJSKi4o/?t=1655) self-play that's his version and then what happens is when you map out every single permutation when you go and play keith who's the best chess player in the world what you're doing at that point is saying okay keith made this move so you search for what keith's move is and you have a distribution of the best moves that you could make in response response. or vice versa that was the cutting edge approach the different approach which is more you know what people would think is more quote-unquote elegant and less brute force would be jason for you and i to sit there and say okay if keith moves here we should do this we should do this specific variation of the sicilian defense and and it's too much human knowledge and i think what what it turned out was there was a psychological need for humans to believe we were part of the

[28:26](https://realalpha.app/v/KypnjJSKi4o/?t=1706) answer. But what this is showing is because of Moore's law and because of general computation, it's just not necessary. You just have to let go, give up control. And that's very hard for some people. And for others, it's also very hard in some circumstances where a car is driving down the road and it's learning in that process, which is why you need a safety driver. And I think Elon made the right decision to put one in there. Keith, your thoughts? Yeah, a couple of points.

[28:50](https://realalpha.app/v/KypnjJSKi4o/?t=1730) It's not quite that binary, Chamath. I generally agree with your arc. But if you think about LLMs being the most important unlock in AI, LLMs are all trained on human writing. So someone wrote every piece of data that every LLM used, a human wrote at some point in history. So yes, it's true that they've shocked everybody, including OpenAI's original team, on the implications, the broad implications, the general applicability to almost every problem.