Open models can be worth the risk of a closed monopoly

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 28:18

   Some models. So Factory basically routes people to the best model for that task. So you guys both send people to OpenAI and Anthropic, and all the different submodels, but also open source models. People have different feelings about Chinese open source models. Factory obviously uses them. How do you think about the risk that they pose or don't pose, in your opinion? Yeah. So generally, I'm very, very pro open models. I think there is propaganda from BigToken that tries to label, tries to make open models synonymous with Chinese model.

2. 29:01

   Okay. But importantly, these are different things. Now, open models are very, very important. We need those to exist. You know, BigToken doesn't like that because it kind of undercuts their ability to price tokens. But are all the big open models Chinese? So the best ones right now are Chinese, but Nemetron is really good. Okay, Nemetron, I've never even heard of it. It's from NVIDIA, a really good model.

3. 29:24

   It's not at the frontier yet. So I think the existence of open models is really, really important. I think it is a vital interest for the United States to have, and the West more broadly, to have frontier open models. um i also think it's of vital interest to every business in the united states to not be kind of monopoly like have tokens monopolized by the closed model providers um and so there's kind of an interesting dance like on one hand i think the number one most important thing is that there's not just the closed model providers and everything else is not good so i actually think no open models or open models that are from china i will take open models from china for sure because the threat of the only good models being from closed providers is like ultimate monopoly, like the monopoly to end all monopolies. It's the interest of the entire economy for that not to be the case.

4. 30:15

   Now, given that there are open models, I would much prefer the best ones to be from the United States because token share is going to become like just the volume of information that is produced in the world. And if you believe in the West, you should want a lot of those tokens to be kind of representing Western values, even if very subtly. Because yes, if you ask a coding question to, you know, a Chinese model versus a US model, like probably it's not going to be that different. However, these things compound over time. And I think there is a vital interest there. I totally agree. So with like Chinese open models, they're not actually like totally open.

5. 30:36

   So you can't see what they were trained on in ways. There's so much discussion about it on a broad level like just as you just started to go one layer deeper uh like how do you think this would actually like negatively affect americans so i think about it on like a value system yeah like i think the way that people interact with models over a really long time it becomes such a reflexive relationship because if you start using these like actually at 10 years old you're absolutely right like literally like the language that people sort of thought that does it affect like your day-to-day language you're absolutely writing it's like people do come for advice they come for work all these things it really truly like subtly shapes someone view on the world there's almost never been a technology that you have this like one-on-one relationship with

6. 31:20

   that is like so constantly feedback looping yeah even if we look at the most like malicious uh domain for this as it relates to what we do in like software so on one hand um if you host chinese open models in u.s you know data centers they're not going to see your data explicitly But there are still a couple of kind of dangerous vectors where something can happen. So one would be like the sleeper agent category of this, which is basically, you know, somewhere in the latent space of this model.

7. 31:52

   If a certain word or a certain trigger happens, it's going to go and try to exfiltrate your data and send it to some endpoint that's maybe in some random. In Europe or Africa, that's then going to go to China. So that's one thing where nominally, it seems like it's fine. But then on some trigger or some activation, it behaves adversarially. That's one thing. And you think that's possible? Certainly, it's possible. It's certainly possible. There are papers that explore this. And it's very difficult to test for. We should very much test for this. But it's pretty difficult to do so thoroughly. That's one.

8. 32:21

   the second is um the uh the code that it generates in the west could be worse code and this can happen actually intentionally or unintentionally so the way this can happen intentionally is like you know in its latent space you make the model associate let's say your documentation and your code base is in english you can make it associate english documentation with like code that has zero days or like big vulnerabilities you can do this intentionally where it's such that you know and it happens super super rarely like they're crazy correlations you can make such that this is really hard to detect but you can make it such that if it's you know working in a in a an american environment or you know western code base that every so often it'll introduce these like very nuanced vulnerabilities this can be done like intentionally or it can even be by accident

9. 33:11

   which is like if you're making a model in china you're going to make sure it's really good in the context of like Chinese codebases, your data just might be less robust for like English or, you know, whatever other languages. So this can happen intentionally or unintentionally. But that's another big failure mode. And I think with both of these, it can happen over such a long time horizon that it could be hard for us to detect now.

10. 33:33

    So that is kind of like the risk area, which certainly exists. But again, I would take that risk over the risk of like a monopoly to end all monopolies in the United States.