Human-generated data is running short, so AI needs synthetic data

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 32:09

   Well, there's other various elements that decide competitiveness for large AI. There's for sure the talent of the people matter. The scale of the hardware matters and how well you're able to bring that hardware to bear. so you can't just order a whole bunch of gpus and if they don't then you can't just plug them in so you've got to you've got to get a lot of gpus and have them um trained trained coherently and stably um then it's like what unique access to data do you have i guess distribution matters to some degree as well like how do people get exposed to your ai those are those are critical factors for if it's going to be like a large foundation model that's competitive um You know, as, as many have said, I, my friend Elias has said, you know, we've kind of run out of pre training data for human generated pre, like human generated data, you run out

2. 33:09

   of tokens pretty fast, certainly of high quality tokens. And, and then you and you have to do a lot of you need to essentially create synthetic data um and and be able to accurately judge the synthetic data that you're creating to verify like is this real synthetic data or is it an hallucination that doesn't actually match reality um so achieving grounding in reality is is is tricky but but we're at the stage where there's more effort put into synthetic data um and like right now we're training grok 3.5 which is a heavy focus on reasoning going back to your physics point uh what i heard for reasoning is that uh hard science particularly physics textbooks are very useful for reasoning whereas um i think researchers have told me that uh social science is totally useless for reasoning uh yes that's probably true um so yeah um there's something that's going to be very