The alpha, twice a week
That was 4 min.
Get the best new moments in your inbox every Wednesday and Saturday. It's free.
Check your inbox
We sent a link to your address. Click it to finish signing up. Not there in a minute? Look in spam.
or
Minus One · 4 min · Dec 13, 2024
Model progress now depends on specialized human data
Alexandr Wang says public data and pretraining are hitting limits, while post-training relies on specialized, high-quality datasets. He expects data production to scale alongside compute, not be replaced by synthetic data alone.
Alexandr Wang is founder and CEO of Scale AI.
Dropped out of MIT to co-found Scale AI in 2016; Meta invested $14.3B in it in 2025 and hired him to lead Superintelligence Labs.
Aditya Agarwal, Host · General partner, South Park Commons
One of Facebook's first engineers, he helped build its first search engine; later CTO of Dropbox.
About this segment
Alexandr Wang says public data and pretraining are hitting limits, while post-training relies on specialized, high-quality datasets. He expects data production to scale alongside compute, not be replaced by synthetic data alone.
From “Scale AI & the Startup Hunger Games with Alexandr Wang”, Dec 13, 2024. Watch the full episode
Answers “What will AI look like in the next few years?” See all 84 answers
Answers “Is AI progress slowing down?” See all 15 answers
Thanks. We'll look at and fix what's off.
“we're hitting the limits of all publicly available data.”
Alexandr Wangat 31:12 · Minus One31:12Takeaways
- Publicly available data is approaching its limits.
- Synthetic data helps, but doesn't replace real data's richness.
- Data production needs to scale alongside compute.
In this episode
Full episode- 34:31AI founders need to separate conviction from industry hype4:18
- 42:08AI startups can win by focusing where giants cannot3:13
30:15Model progress now depends on specialized human data3:55
- 5 more moments in the full episode
Transcript
Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.
- 30:18
one of the biggest limitations to building the next kind of generation of frontier models is simply accessibility of like data. How real do you think the data wall is? What's your take on kind of essentially synthetic data for training models? And obviously, I think in this overall conversation scale is a huge kind of like you know uh huge part in the conversation so curious to get your thoughts there yeah i think like i think the the um the frenzy of the last you know 12 to 18 months of conversation was just like who has more compute who has more chips you know the more chips you have you know you're going to win and so it was just this like very un-nuanced conversation around who has the biggest cluster and whoever's the business cluster is going to win and then i think what's like coming to light is, you know, even with a much bigger cluster, you, you were
- 31:12
hitting some data limits, like, you know, we've, we're hitting the limits of all publicly available data. And, you know, we need a tandem approach of, you know, production, you know, what are the specialized data sets in addition to the computational power that's going to yield much greater performance and so i think i mean the this the uh the the data wall and the sort of like the wall on progress that we've hit right now is like certainly real you know i think we've hit some kind of pre-training limits um and a lot of the progress now is coming from post-training that post-training is much more bottlenecked by specialized data sets and and uh and sort of high quality data sets that don't look like the ones that we have on the internet. So data is really critical. I think there was a sort of like, there was a
- 32:01
One of the research beliefs was that, oh, the models will just generate large-scale reasoning traces or other kind of data that we'll just pre-train the models on. That hasn't really panned out. I think a lot of the experiments on synthetic data, it just turns out you lose so much of the real richness in the data distribution if you use synthetic data. So I think the reality of what's going to happen going forward is like, you know, we're going to need to rely on new forms of human generated data to get to where we want to go with the models.
- 32:32
But if we scale data, in addition to scaling compute, we can keep making progress. We may not, it may not be necessary to have this sort of like, you know, trillion dollar, five trillion dollar, ten trillion dollar clusters that we were talking about a year ago. But we still probably need the hundred billion dollar clusters. So I think it's kind of, I think there's some kind of normalization to the conversation that's happening.
- 32:57
Yeah, it's kind of like, it was actually all about who had the biggest cluster and then will capitalism work and then we actually have a lot more chips, which is great. But I do think that the new bottleneck is on the data side. So, I mean, I'm curious, how do you think what scale's role in that? Like, obviously you guys play a big role in the post-training part of the world, but do you think that, are there opportunities for scale to even ramp up its kind of its presence on that front?
- 33:23
Yeah, I mean, we're having to ramp up data production across the board. So I think that, like, I think that, frankly, what's going to have to happen is, like, if you zoomed all the way out on the, like, path to AGI, so to speak, I think we're going to need, you know, the computers that have to scale exponentially, and then you'll have this, like, smaller curve, but this equivalent curve of, like, data production needing to scale up exponentially.
- 33:45
And so I think we just need to sort of, like, keep scaling up on that curve. and it needs to scale with everything else. So you can't just scale the data ahead of the computer. You can't get, one can't get too far ahead of the other. But yeah, I think our role is really, our key role to play is in producing all the data that actually enables to get to that, get to data scaling in addition to compute scaling.