String-focused post-training can damage probability calibration

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 54:09

   and i guess i imagine this is circling back on rlcd um but like historically uncertainty estimation has been like very difficult um and you know particularly when we are talking about these you know large compressed versions of the internet that historically generate text and like uh and large complex models generally like so how do you how do you do that hard work blood sweat and tears and also and also like you know it's hard work like we were in stealth for a long time you know being two years in stealth is like kind of insane in in like ai terms the amount of i'm not asking you to divulge the family jewels it's more like is it is it grounding on something like What's the guiding star, the like the principle that kind of helps us?

2. 55:10

   Understand why we should trust these probabilities as opposed to, well, they've just got some token generator that's been trained to generate numbers between zero and one. I'm going to give you like one more bone in the last thing, which is, you know, we did work in stealth for a long time. We missed so many fads, even though we thought Jev would have been great for a lot of those fads. Like, holy shit, it would have been cool to launch Jev at the same time as OpenClaw. It would have been awesome because everyone has like a lot of reliability issues And also, these compressed things of the internet are, as far as we can measure, decently calibrated.

3. 55:46

   They're not perfectly calibrated, but they are way, way more calibrated than RLHF models are. Actually, RLHF and RLVR destroy the calibration of the models immensely, because in order to output strings well, you actually need extreme overconfidence. It requires a mode-dropping loss in order to make that work, which just completely warps the probabilities and i think that the gpt4 paper started talking about this because that's when models started not shipping their pre-trained versions and only started shipping post-trained versions so i think the gpt4 paper went into it a bit but the models used to be decently calibrated look at a lot of opening um anthropics early publishing you know measuring uh with what they called pino the probability it knows facts like these are all like pretty decent and it's definitely not good enough i would guess jeb

4. 56:38

   would crush all of those like super easily the the thing is that there's ways to bootstrap calibration and reliability and intelligence from all of that the same way that rlvr bootstraps reasoning right like in an ideal world the reasoning of rlvr doesn't come from humans writing the reasoning because it does it differently but if you have the right rl objective of calibrated decisions there are ways to get that working from that seed data into becoming more and more and more calibrated over like the training that i don't really know what it's called epochs training stages whatever so there are there's like hand wavy ways that makes sense that you can get calibrated stuff out of not quite nothing but out of like the beautiful raw material that AI has today, because AI has like a lot, like, I feel like AI is still a diamond in the rough.

5. 57:33

   You know, there's so much stuff to do there. And if I wasn't doing GEV, I would probably be ideally not starting a company, but doing other cool stuff on top of the AI models.