The alpha, twice a week
That was 2 min.
Get the best new moments in your inbox every Wednesday and Saturday. It's free.
Check your inbox
We sent a link to your address. Click it to finish signing up. Not there in a minute? Look in spam.
or
Possible · 2 min · Oct 7, 2026
Open source makes safety tools a one-way door
Camille François describes OpenAI releasing a moderation model under Apache 2, then seeing it adopted and tested. Open licensing lets others keep using it even if the company changes course.
Camille François is a founding president of ROOST and professor at Columbia University.
Reid Hoffman, Host · Co-founder, Manas AI
Co-founded LinkedIn; one of OpenAI's first investors; Greylock partner.
- AF
Aria Finger, Host
About this segment
Camille François describes OpenAI releasing a moderation model under Apache 2, then seeing it adopted and tested. Open licensing lets others keep using it even if the company changes course.
From “Building safety tools no one can take away with ROOST’s Camille François”, Oct 7, 2026. Watch the full episode
Thanks. We'll look at and fix what's off.
“Open source is a one-way door, right? So when OpenAI says we are actually going to share this model under an Apache 2 license”
Takeaways
- Apache 2 licensing means the company cannot pull back the released model.
- Wikimedia adopted the shared moderation model.
- A hackathon let others test and challenge it.
In this episode
Full episode- 17:02Discord turned a vendor shutdown into shared safety infrastructure2:22
- 24:58Safety teams need tools for investigation and review, not just detection3:25
- 51:42A two-person app used open-source tools to report 19 cases2:59
- 8:17Social media was a detour from tech’s open foundations3:00
11:10Open source makes safety tools a one-way door2:26
- 29:42Shared safety tools can help startups compete4:51Up next
- 35:41Shared tools can be open without exposing every detection rule2:45
- 46:28Adding safety to AI apps is still a multi-step process2:50
Transcript
Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.
- 11:15
I think some of those leaders are building this new open source foundations with us. So last year, for instance, we did a big effort with OpenAI. Our team worked closely with their scientific and engineering team to take one of the core piece of their safety stack, a model that was doing a lot of moderation behind ChatGPT and then Sora, and to actually share it with everybody, to put it out in open weights.
- 11:44
Now, that's significant for a number of reasons. But the first one is prior to that, it's not like you didn't see big companies share safety innovation. But a lot of that sharing looks like, oh, we're going to make this available to everybody. Usually that lasts for the duration of a really good marketing campaign. And then, of course, like, you know, priorities change and then people move on. And so things are not maintained.
- 12:08
People are struggling to get access. Things sort of fall behind. And then, of course, at any moment, they can pull the rug, right? They can say, actually, we changed our mind. But open source isn't that. Open source is a one-way door, right? So when OpenAI says we are actually going to share this model under an Apache 2 license and we're going to put it on hugging phase, that means if tomorrow they change their mind, they can't get it back.
- 12:32
And so people know they can build on it, right? So we did that. And they made that statement at the time saying that they believe everybody should be free to study, modify, and use critical safety techniques. It's kind of fun to see OpenAI quoting Richard Stallman. But I love that. And after that, we also did a really nice hackathon working with both OpenAI and HuggingFace. So we invited people saying, OK, look, this model has been shared.
- 12:58
Come test it. And so we saw people who are really exciting to immediately use the model. For instance, today, this model is used by Wikimedia, right? We also keep in mind that large, important tech nonprofits also deserve safety tools. But we also had a bunch of people come and break the model, which was really fun to see. I remember there's a team who I think was building recipes for meth and ecstasy in Latin.
- 13:24
So, yeah, that's sort of like what we, you know, we see some of those large lab being like, OK, yes, we think we want, you know, we want to tackle safety a bit differently now.