Research prototypes can work without frontier-scale compute

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 36:36

   be expecting in the community so one consequence of the age of scaling is that there was this um scaling sucked out all the air in the room yeah and so because scaling sucked out all the air in the room everyone started to do the same thing we got to the point where we are in a world where there are more companies than ideas by quite a bit actually on that you know there is the silicon valley saying that says that ideas are cheap execution is everything and people say that a lot yeah and there is truth to that but then i saw i saw someone say on twitter something like if ideas are so cheap how come no one's having any ideas and I think it's true too I think if you think about research progress in terms of bottlenecks there are several bottlenecks if you go back to the and one of them is ideas

2. 37:45

   and one of them is your ability to bring them to life which might be compute but also engineering so if you go back to the 90s let's say you had people who had pretty good ideas and if they had much larger computers maybe they could demonstrate that their ideas were viable but they could not so they could only have very very small demonstration and not convince anyone yeah so the bottleneck was compute then in the age of scaling computers increased a lot and of course there is a question of how much compute is needed but compute is large so compute is large enough such that it's like not obvious that you need that much more compute to prove some idea like i'll give you an analogy alex net was built on two gpus that was the total amount of compute use for it the transformer was built on eight

3. 38:44

   264 GPUs. No single transformer paper experiment used more than 64 GPUs of 2017, which would be like what? Two GPUs of today? So the ResNet, right? Many, like even the you could argue that the 01 reasoning was not the most compute heavy thing in the world. So they're definitely for research you need, like definitely some amount of compute, but it's far from obvious that you need the absolutely largest amount of compute ever for research.

4. 39:22

   You might argue, and I think it is true, that if you want to build the absolutely best system, if you want to build the absolutely best system, then it helps to have much more compute. And especially if everyone is within the same paradigm, then compute becomes one of the big differentiators. yeah i guess well it was possible to develop these ideas i'm asking you for the history because you were actually there i'm not sure what actually happened but it sounds like it was possible to develop these ideas using minimal amounts of compute but it wasn't the transformer didn't immediately become famous it became the thing everybody started doing and then started experimenting on top of and building on top of because it was validated at higher and higher levels of compute Correct.

5. 40:07

   And if you at SSI have 50 different ideas, how will you know which one is the next transformer and which one is, you know, brittle without having the kinds of compute that other frontier labs have? So I can comment on that, which is, the short comment is that, you know, you mentioned SSI. Specifically for us, the amount of compute that SSI has for research is really not that small. And I want to explain why.

6. 40:42

   Like a simple math can explain why the amount of compute that we have is actually a lot more comparable for research than one might think. now explain so ssi has raised three billion dollars which is like not small but it's like a lot by any absolute sense but you could say but look at the other companies raising much more but a lot of what their a lot of their compute goes for inference like these big numbers these big loans it's earmarked for inference that's number one number two you need if you want to have a product on which you do inference you need to have a big staff of engineers of salespeople a lot of the research needs to be dedicated for producing all kinds of product related features so then when you look at what's actually left for research the difference becomes a lot smaller now the other thing is is that if you are doing something

7. 41:45

   different, do you really need the absolute maximal scale to prove it? I don't think that's true at all. I think that in our case, we have sufficient compute to prove, to convince ourselves and anyone else that what we're doing is correct. There's been public estimates that companies like OpenAI spend on the order of five, six billion dollars a year, even just so far on experiments. This is separate from the amount of money they're sending on inference and so forth. So it seems like they're spending more a year running research experiments than you guys have in total funding.

8. 42:22

   I think it's a question of what you do with it. It's a question of what you do with it. Like they have, like, is the more, I think in their case, in the case of others, I think there is a lot more demand on the training compute. There's a lot more different work streams. there is there are different modalities there is just more stuff and so it becomes fragmented