OpenAI bet on deep learning and scale before the field agreed

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 13:45

   And one of the things that worked really well in retrospect was we said from the very beginning we were going to go after AGI at a time when in the field you weren't allowed to say that because that just seemed impossibly crazy and, you know, borderline irresponsible to talk about. So that got his attention immediately. It got all of the good young people's attention and the derision, whatever that word is, of the mediocre old people.

2. 14:12

   and I felt like somehow that was like a really good sign and really powerful and we were like this ragtag group of people I mean I was the oldest by a decent amount I was like I guess I was 30 then and so you had like these people who were like those are these irresponsible young kids who don't know anything by anything and they're like saying these ridiculous things and the people who that was really appealing to I guess are the same kind of people who would have said like it's a you know I'm a sophomore and I'm coming or whatever and they were like let's just do this thing let's take a run at it and so we kind of went around and met people one by one and then in different configurations of groups and it kind of came together over the course of in fits and starts but over the course of like nine months and then it started i mean

3. 14:57

   and then it started started happening and one of my favorite like memories of all of open ai was ilia had some reason that google or something that we couldn't start in we announced in December of 2015 but we couldn't start until January of 2016 so like January 3rd something like that of 2016 or like very early in the month people come back from the holidays and we go to Greg's apartment maybe there's 10 of us something like that and we sit around and it felt like we had done this monumental thing to get it started and everyone's like so what do we do now and what a great moment it reminded me of when startup founders work really hard to like raise around and they think like oh i accomplished this we did it we did it and then you sit down and say like fuck now i gotta like figure out what we're gonna do it's not time for popping champagne that

4. 15:48

   was actually the starting gun and now we gotta run yeah and you have no idea how hard the race is going to be it took us a long time to figure out what we're gonna do um but one of the things that I'm... Really amazingly impressed by Ilya in particular, but really all of the early people about, is although it took a lot of twists and turns to get here, the big picture of the original ideas was just so incredibly right. And so they were like up on like one of those flip charts or white boys, I don't remember which, in Greg's apartment. And then we went off and, you know, did some other things that worked or didn't work or whatever, and some of them did, and eventually now we have this like system. And it feels very crazy and very improbable looking backwards that we went from there to here with so many detours on the way, but got where we were pointing.

5. 16:41

   Was deep learning even on that flip chart initially? Yeah. I mean, more specifically than that, like do a big unsupervised model and then solve RL was on that flip chart. One of the flip charts from a very, this is before Greg's apartment, but from a very early offsite. I think this is right. I believe there were three goals for the effort at the time. It was like, figure out how to do unsupervised learning, solve RL, and never get more than 120 people. Missed on the third one. But the predictive direction of the first two is pretty good. So deep learning, then the second big one sounded like scaling. Like the idea that you could scale. That was another heretical idea that people actually found even offensive.

6. 17:26

   You know, I remember a rash of criticism for you guys at that moment. When we started, yeah, the core beliefs were deep learning works and it gets better with scale. And I think those were both somewhat heretical beliefs. At the time, we didn't know how predictably better it got with scale. That didn't come for a few years later. it was a hunch first and then you got the data to show how predictable it was but people already knew that if you made these neural networks bigger they got better yeah um like that was we were sure of that yeah um before we started and what took the like word that keeps coming to mind is like religious level of belief was that that wasn't going to stop everybody had some reason of oh it's not really learning it's not really reasoning it can't really do this it's you know it's like a parlor trick and these were like the eminent leaders of the field

7. 18:21

   and more than just saying you're wrong they were like you're wrong and this is like a bad thing to believe or a bad thing to say it was that there's gotta you know this is like you're gonna perpetuate an AI winter you're gonna do this you're gonna do that and we were just like looking at these results and saying they keep getting better then we got the scaling results it just kind of breaks my intuition even now and at some point you have to just look at the scaling laws and say we're going to keep doing this and this is what we think it'll do and it also it was starting to feel at that time like something about deep learning was just this emergent phenomenon that was really important and even if we didn't understand all of the details in practice yet which obviously we didn't and still don't that there was something really

8. 19:12

   fundamental going on it was the pgism for this is we had like discovered a new square in the periodic table yeah and so it we just we really wanted to push on that and we were far less resourced than deep mind and others and so we said okay they're gonna try a lot of things and we've just got to pick one and really concentrate and that's how we can we can win here which is totally the right startup takeaway and so we said well we don't know we don't know we do know this one thing works so we're going to really concentrate on that and I think some of the other efforts were trying to outsmart themselves in too many ways and we just said we'll just we'll do the thing in front of us and keep pushing on it. Scale is this thing that I've always been interested in at kind of just the emergent properties of scale for everything for startups.