Small teams can win where general models stay weak

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 25:36

   So the thing about Google is you co-design everything on the system from the processors to the products. Which are the layers that someone like Google would keep building and compounding and being better, and where does a two, three-person team can still win? Yeah, I mean, I think obviously Google and our Gemini models and our hardware infrastructure are really trying to build very general models that can do almost anything.

2. 26:07

   But in a lot of cases, that means that we don't have a lot of attention on particular domains where perhaps a really well-designed surface that and maybe a model and set of skills or maybe a specialized model that isn't in sort of the general mix of things that our models do well can actually have a significant advantage because you can build something delightful and you know really high accuracy here really high quality for a domain that you are really passionate about and I think that's that's where you know the two or three people in a room, uh, building that, that they're really excited about can have an advantage. Um, but I, I would also caution that the general models are definitely getting better at a broader and broader range of things. So you have to figure out, you know, is that thing you're

3. 27:03

   working on, is that going to be a durable thing? Or do you think the models, uh, at the forefront are going to get better at that in the next six months or 12 months, or is it something they're not going to be able to do for a couple of years or three years. And, you know, you want to weigh that as you're, as you're deciding what to work on. So let's dive deeper into this. So the general models, of course, you're going to keep working on and keep making them all better. And how should the audience reason about what are those areas that it doesn't? I mean, how should founder think about things to pick on and work on yeah i mean i mean the most important thing is to pick something you're super excited about and want to build and you think would be useful in the world right so if you do that um that that's you're already way ahead uh than if you wake up and you're like oh i

4. 27:55

   don't really want to do this or whatever or you're going to build something that is actually not that useful to the world or to many people. So, I think that's the number one selection criteria. I try to apply for what problem should I work on next. Second, I think you want to look at what the current, more general models can do in that problem domain. You can test them with like, are they able to do this thing very well?

5. 28:25

   And if they're completely failing, that's probably a good sign. If they're kind of able to do some of it but not very well, that's maybe not a great sign because that's probably a sign that the capability is starting to be present in those models. And with more training data or larger scale models or whatever, it's likely to get better. So, you know, look for something where the model succeeds 0% or 1% of the time, not 20%.

6. 28:54

   How do you find those? I mean, are those things effectively out of distribution from the training set? And what exactly is the problem shape that fits that? Yeah, I mean, I think sometimes it's a product that you build that might have access to particular kind of data that the underlying model might not, the general model. So it might be you're building something to help users organize all their own personal information.

7. 29:21

   And the model won't necessarily have access to that. And so there you can have a big advantage because all of a sudden your model has visibility or your product has visibility into important data. It could be some incredibly hard problem where if you get the right training data and you can train a more specific model than a general purpose one, you can actually do that in a very affordable way. Maybe it doesn't take that much compute to train a niche model for this particular problem, but you can get something that's highly accurate.

8. 29:54

   that can sometimes be a really good uh building block for for solving a important problem that is maybe not handled very well by the general model i think that's interesting i think you're basically two paths the first path is uh a little bit funny is uh you guys are organizing the world's information yeah that's probably kind of well covered yeah but organizing your personal information that's open which is funny yeah and then the second path um you talked about more specialized models in certain domains can you tell us more about what are some of these domains yeah i mean i think like if you look at uh my colleagues work on say alpha fold that was a very specific model for uh protein folding and it was highly successful um and was able to really handle that domain quite well, so that all of a sudden you now have this amazing tool

9. 30:47

   and model that can give you answers to questions about proteins and their structure really effectively. But it's not a general model. It's a very specific one. And there are other, I think, domains where that kind of approach can work really well, maybe in material science or chip design or things like that, that will enable you to leverage the capabilities of a very