Building safety tools no one can take away with ROOST’s Camille François

No, yes. Of course, I recognize that description. I do think that I am able to work with and within institutions. But yes, I like the energy of doing something together in a collective, maybe with a little bit of a pirate spirit. And when we were preparing for our very first ROOST team retreat, I actually found a book, which is great, called Be More Pirate. that looks back at the history of the glorious days of piracy and think through, OK, there are institutional lessons to take from those days. And there's a way to rethink about that moment in time as a moment in time that brought a lot of advances in equality and democracy.

And so maybe there's a different lens we can apply to our understanding of piracy. But jokes aside, I think that I've always been part of the open source and free software movement. When I was a student, I founded Students for Free Software France and then sort of served on the board of the global organizations, actually how I met my husband. And that organization no longer exists. But back in the day, we were referred to it as Baby Pirate Hacker Club. So, yes, I can see where that comes from. Arr. Yeah, exactly. I mean, anti-authoritarianism. I'm with you. That's great. And it seems like everyone is talking about online safety, privacy.

You were way ahead of the curve on this. You were building ROOST, you mentioned, which is a nonprofit that makes online safety tools free and open source. For someone who is hearing about this for the first time, who are these tools for and what do they help people do? Yeah, they're for everybody. And ROOST is a nonprofit. It's also a movement. And so we do this with a lot of people. We do this with large AI labs, but we do this with small builders. We do this with philanthropic organizations. We do this with many different types of institutions. And essentially, if you think about safety, we have at least three problems. The first one is, I think it's fair to say that tech and AI has progressed faster than our ability to keep people who interface with these technologies secure, not only when it comes to kids and children.

And so we have an innovation gap, right? We need to fast forward our innovation and progress on safety tech. The second problem that we have is sort of a distribution gap, right? There are a lot of safety tools that are OK, but people don't have access to them. And there are many more builders who are building new applications, new websites, and who want access to the tools to do things right, to protect their users, to protect their communities, to do safety by design, right? That's a real problem that we have. It's nonetheless a good news, right? That means there's more appetite for safety. And then, of course, the third problem is there's a little bit of a crisis of trust when it comes to safety technology.

It's because it has long been very opaque, right? So people tell you like, hey, we're using technology to keep you safe. And you're like, okay, what is it? Can I see it? And it's like, no. And so, of course, we can change that model. Here, open source really is a method, right? where we can accelerate innovation, make sure everybody who wants those technologies have access to it. And then, of course, where we can say, hey, here it is. Here's a safeguard. Go audit it. Modify it. If you don't think it works for your use case, go make it work in your language, your context. And then if you make a better version of it, just go distribute it or sell it, right, for that matter.

You've spent a number of different years, both as a pirate entrepreneur and a professor, and then also years investigating online harm, work on informing the Senate on disinformation. Was there a moment that convinced you to build ROOST? Was there a kind of a, ah, this would be a way of changing this ecosystem? And what was that moment? And how did your experiences lead you there? I don't think there was a moment. I think that has been slowly building up. And frankly, not just for me, but for the field. And if I think about my own trajectory in safety, of course, I have taken from all those different experiences, ingredients that you find in ROOST, right?

So when I was studying information operation, one of the things that you very quickly find is information operations are cross-platform, right? And so you find them spreading on smaller bulletin board, weird little forums, apps that nobody's ever heard of. And when you sort of unravel this, you have a lot of people coming in saying, oh, we did not want to use the school board in this way. Help us. Right. And so you sort of understand that any sufficiently smart and motivated adversary is going to take advantage of the fact that there are many surfaces and smaller surfaces and not everything is protected.

Of course, that matters a lot for the theory of change at ROOST. When I was at Niantic, that was a really formative experience because I had a great CEO, John, who was very supportive of our safety efforts. And of course, Niantic had Pokemon Go. A lot of kids play Pokemon Go. And when we said, OK, we're going to build a really good, robust infrastructure for safety, I went and I sort of knocked on doors and looked around. I'm like, what's the state of the art? And I did not like what I found. And the state of the art was like, yeah, what we do is we kind of everybody builds their own half-baked stuff.

And then you buy a few good vendors and then you duct tape everything together. And that's what everybody does everywhere. So I was like, OK, well, that sounds nuts. It has to be a better way. I was coming from more of a cybersecurity background where, of course, there's a different approach, right? Like there is a shared infrastructure of open source tools across the field so that you can focus your resources on building the piece that makes sense for you. But you don't have to sort of fully reinvent everything from scratch all the time. And so I started thinking about, OK, well, let's try to figure out if those tools, everybody's sort of cobbling together, are they the same for everybody?

Like, is it are we doing this because people have such unique needs? And so I started interviewing a lot of safety engineers across the field. And they were like, absolutely, they're the same. Most of the infrastructure is steady. So we did this project to sort of document what are the core components of safety infrastructure. We published a piece called the Dyer Framework that goes for detection, investigation, review, and enforcement. But really what engineers were telling us is, yeah, we get staffed on something. We built 30% of a workable thing. And then we get assigned to another project.

And so everybody's sort of constantly rebuilding basic foundations, which is really frustrating. And so we said, OK, great. There's an opportunity to change the way we do safety in tech, to sort of rewire the way we do safety in tech. I feel like everyone's talking about there's a lot of other issues and discussions around AI. But a lot of people are seeing AI as sort of social media 2.0. We got social media. We were so excited. There were these harms, some of them to children, some about misinformation that we feel like we didn't sort of attack soon enough. and people are worried that it's going to happen with AI.

Like, do you think that if ROOST had been around 10 years earlier, the landscape would be different? And like, where do we go from here? I actually think about social media as an exception to the rule of how we build tech, as a blip in the timeline. That's just a bizarre happenstance, right? Because if you take the long arc of tech, the way we built the internet,

Camille Françoisthe way we built the foundation of tech around us is on open protocol and on open source software,

right? TCPIP was an open protocol. Tim Berners-Lee donated the web to humanity, right? And so the reason why we can innovate and then keep tech relevant and keep tech responsive to our needs as a species is because we do this in the open. And then there's a sort of small era of social media where we kind of forget about all these lessons. And it's bizarre, but it's a fact that we've inherited all of the safety tech that we have from that blip in the timeline, from that era. That's why it's brittle. That's why it's hyper-centralized. That's why it's opaque. And so what we're doing with Roots, frankly, is sort of bringing back all those normal principles of how we build good tech into the world of safety.

We're not really inventing something new rather than sort of like getting back to the roots of the internet. Exactly, right? Changing the approach to something we know works, which is, okay, if you want faster Ask collective innovation. If you want people to have access to the tools they deserve, if you want people to trust the things that they run their lives on, give them good shared open source foundations. And do you think the leaders of the Frontier Labs are learning from any of those mistakes or like what mistakes are they still repeating? I think some of those leaders are building this new open source foundations with us.

So last year, for instance, we did a big effort with OpenAI. Our team worked closely with their scientific and engineering team to take one of the core piece of their safety stack, a model that was doing a lot of moderation behind ChatGPT and then Sora, and to actually share it with everybody, to put it out in open weights. Now, that's significant for a number of reasons. But the first one is prior to that, it's not like you didn't see big companies share safety innovation. But a lot of that sharing looks like, oh, we're going to make this available to everybody. Usually that lasts for the duration of a really good marketing campaign.

And then, of course, like, you know, priorities change and then people move on. And so things are not maintained. People are struggling to get access. Things sort of fall behind. And then, of course, at any moment, they can pull the rug, right? They can say, actually, we changed our mind. But open source isn't that.

Camille FrançoisOpen source is a one-way door, right? So when OpenAI says we are actually going to share this model under an Apache 2 license and we're going to put it on hugging phase, that means if tomorrow they change their mind, they can't get it back.

And so people know they can build on it, right? So we did that. And they made that statement at the time saying that they believe everybody should be free to study, modify, and use critical safety techniques. It's kind of fun to see OpenAI quoting Richard Stallman. But I love that. And after that, we also did a really nice hackathon working with both OpenAI and HuggingFace. So we invited people saying, OK, look, this model has been shared. Come test it. And so we saw people who are really exciting to immediately use the model. For instance, today, this model is used by Wikimedia, right? We also keep in mind that large, important tech nonprofits also deserve safety tools.

But we also had a bunch of people come and break the model, which was really fun to see. I remember there's a team who I think was building recipes for meth and ecstasy in Latin. So, yeah, that's sort of like what we, you know, we see some of those large lab being like, OK, yes, we think we want, you know, we want to tackle safety a bit differently now. I want to go into a specific area as an example in a second, but let's first come back a little bit. I'm not giving you the recipe of meth and lab. Don't insist. I want it in French. One of the things, like when you think about, like I think there is a lot of fundamental importance about open source. As you know, I was on the board of Mozilla for like a decade and LinkedIn actually open sourced a whole number of projects. And I think it's a generally good thing to do. I think it's also good on certain patterns of safety and security.

And I think the foundation of the internet on it. I don't think it's actually only social media that creates the more turbulent stuff. I mean, we have the mobile internet with Apple now. We have a bunch of, like, there's kind of what's going on behind YouTube, you know, within Google and so forth. So I want you to kind of elaborate a little bit of what does this weave look like and how do we bring, because it's not a disagreement that open source isn't an important thing here to elaborate this point of view, but it's like given a complex environment, what is the role that you see open source doing? And then I'll get to a specific commercial example question. I think here, open source has a unique ability to help us, as I said, reshape how we think about safety, really accelerate innovation, make sure that a lot more builders have access to the tools they want and need, and enable a system that can be scrutinized and be adapted to a lot of little communities. When you think back to the social media era, we talked about the sort of perils of hyper-concentration. There's another peril, which is this idea that I think we held a little bit too long, that we were going to have big global public squares with rules that work for

everybody. Well, I don't think this panned out exactly the way we wanted. I think the reality is you're going to have a lot of smaller communities that have rules that make sense for them. And that's important. That's pluralism, right? That's the world that we want to live in. If I think back about the difference between today, for instance, and say like 10 years ago, as you said, I'm a professor 10 years ago. If I looked at my students' online lives, they were quite concentrated on a few platforms, right? So for instance, they may have like They use, back in the days, what's called Facebook, you know, Facebook for schoolwork and also communicating with family and all that.

But today, I think what we see is people use a lot of different smaller platforms for a number of things, right? They may actually have a WhatsApp group for something, a signal group for something else. They might use a Discord server to coordinate about something and then, you know, whatever, right? So this multiplicity, I think, now of online spaces, both in social and in AI, also means that we need to be able to give all of these communities the tools that they deserve to shape what safety means for them, right? It's also bringing back pluralism, which is something that maybe we had lost sight of a bit during the social media era.

because one of the things I know you've tracked is, you know, Twitter bot Smyte in 2018. I mean, they cut off all the other customers, demonstrated how brittle this is when you have a commercial system. So ROOST is a offset to that in part. What are the things in terms of is ROOST the only one that's doing this? Are there more? You know, is ROOST trying to encourage things like Smyte or other things to be open? Yes. Like, how does the dynamic play out? That's part of, like, the specifics within a commercial environment. Okay. So let's talk specifics of it. Yeah. I love the story of Smyte. So Smyte was a little startup that provided safety technologies to a bunch of companies.

One of them was Discord. And Smyte, which, in fact, is a rules engine, got bought by Twitter because Twitter was like, yeah, that's very helpful for what we do. I get that. And then Twitter shut it down. And so a number of companies, including Discord, was like, oh, we can't protect our users and community.

Camille FrançoisWe just got the rug pulled super, super brutally from under our feet.

And so they did what they had to do, which is they did the very boring work of rebuilding everything from scratch. What they have rebuilt from scratch, they have donated to ROOST. It is the foundation for one of our core software projects called Osprey. Because Discord was like, nobody needs to go through that again. I am going to open source that rules engine. And the second they did that, right, we worked with their engineering team because actually it takes a minute to open source an internal system. BlueSky came in and said, oh, that's great. I want to use that. So they started using this.

And as a result, they started improving this. And so Discord said, OK, well, we want to use the branch that everybody is improving. We want to benefit from the collective innovation of everybody who uses Osprey. And so Osprey is this amazing story of like, this is what the industry learned from Smyte. And this is what Discord decided to donate. And now this project, Osprey, is actually used by many different platforms and organizations at scale to investigate fraud and scams and coordinated harassment. And it's great to see it take a life of its own. Yeah, it's awesome. Yeah, it's also fun for Discord because, well, as I said, like they benefit from the collective innovation and then they were able to spot really good engineering talent too.

And so it becomes fun because all that really good engineering safety work also becomes visible, right? So morale boosting for the people who work on this and then also it's helpful for careers. Yes, absolutely. I mean, every engineer wants to work on something that millions and millions of people are using. That's the benefit of being able to be part of that community. Yeah. And I think it's also, you know, Discord, it's really important to have all the different actors in the ecosystem. You need the researchers, you need the professors, you need the people who are in the companies. And like Discord was uniquely suited to be able to do something because they had had this rug pull. And we talked about you being a pirate. So you've mostly worked sort of from the outside.

That's the theme of the question. I was trying to keep it to you and around. We're only talking about pirates here. But you also were inside of Niantic, as you mentioned. Yes. You know, accountable for outcomes in real time, you know, at an actual company that's pushing deliverables. What did that teach you that sort of the researcher being outside of a company couldn't have taught you about this space? I think it taught me how much tools matter. And if you take a step back and sort of look at the overall conversation that we're having on safety, there's been a lot of discussion on rules, right?

What should be acceptable and not acceptable. And I understand that focus because I think roles are really the most visible part of the pyramid, right? It's the thing that we can all look at and sort of negotiate together. I think I learned at Niantic that policy without tools is basically poetry, right? It looks great. You can recite it, but you can't really protect people with it concretely. And so the importance of those systems, the importance of infrastructure, the importance having multiple systems who you know will be interoperable and will work together, because a lot of important context that you need to protect people also get sort of lost in translation when there's a handoff between one process and another process, maybe one internal tool to another internal tool.

I think that I got a unique sort of very up-close and personal perspective when I built a trust and safety function. Actually, tooling is really important, and it's one of the things that academia usually doesn't get because it's not quite the way that academia operates. So that's an excellent bring into the academic environment, generally, not just in safety, not just in open source, not just in… And, you know, RIS was incubated at Columbia University, right, at the Institute for Global Politics. I think we forget that academia can also incubate amazing tech nonprofits, right? We look at big business schools and all the startups they incubate.

But there's actually a nice tradition of amazing tech nonprofits being incubated out of academia. I think of, like, Scratch, Love Scratch, coming out of the MIT, right? So, like, universities here have a role to play. And, yes, if they focus on boring things like tooling, we can go really far. It will help. It will help. So I saw that your ABC framework was cited in Anthropik's report on AI MSUs. So, one, a first, briefly tell us about the framework. And then the question is, does generative AI break the premise of the ABC framework? Because now that you can have a single actor, you know, synthesized actor behavior content all at once, does it work or does it need to be updated?

So first framework and then that question. The ABC framework has had a fun life since it was initially published, I think, in 2019. Initially, I was looking at information operations and then sort of more broadly at moderation. And ABC here stands for actors, behaviors, and content. And it simply says that essentially trying to figure out who are undesirable actors on a platform, for instance, convicted child sexual offenders, what are manipulated behaviors on a platform, and what is undesirable content. It's actually three different exercises and require three different approaches that will draw on different disciplines, right?

One will be closer to borrowing from spam and the other will be closer from borrowing to cybersecurity. So it was just trying to sort of put some order around those three things that we tend to conflate and focus on content where you can see problems where there is both a problematic actor and manipulative behavior and undesirable content. But also, again, it's helpful to sort of separate those three things and think about the tools and approaches that are appropriate for each. It was very helpful in social media days to help sort of reorganize moderation functions where a little bit of the cybersecurity mindset would also come in.

And it was interesting because OpenAI, I think, were the first ones to also in the sort of AI era to also cite the ABC framework. and my talented colleague Ben Nemo, who's at OpenAI, said, actually, I think we can do completions for C. And so he said the ABC framework still works in the context of generative AI. It's just like actor behaviors and completions. Sometimes, you know, the AI will complete something and you're like, that is undesirable. And so I think he's the one who sort of like brought it into the AI context. And I was also very pleased to see it cited in the anthropic report, yeah.

Well, so if we're digging into sort of this safety work, you have detection, investigation, review, enforcement. Which of those is hardest to get right? And we're – so one, which of those is hardest to get right? And then we're talking about perhaps it's most important for there to be safety among the big actors that are serving millions, tens of billions, billions of people. But there's a long tail. And so that's also the problem that smaller actors can't – you know, they're under-resourced. And so when the under-resourced folks come in, what do they skip first in those four things? If you think of detection, investigation, review, and enforcement as sort of a pyramid, right, or an iceberg, rather, the exposed part, the thing that everybody sees and likes to comment on is detection, right?

So the question is always like, oh, do we know how to detect this harm? It's a helpful, fun conversation. I'll get back to it in a second. But in fact, once you've detected a harm, there are a number of things you have to do to be able to take action. And at ROOST, we focused on sort of building those boring tooling infrastructure pieces so that people have the means to take actions when something is detected. So we talked about Osprey. Osprey is an investigation tool. tool. It allows you to say, oh, I can describe an activity that I don't want on my service by its pattern. Please, real time, go find a pattern and help me take action. Super helpful, for instance, if you want to root out fraud or if you're dealing with sadistic online exploitation and you have a signal that you want to be able to detect at scale. So that's Osprey. And as I was sharing, this is coming from Discord.

The other big tool that we have and that people are using is called Coop. Coop tackles review. Now, Coop came into a ruse through also a bit of a bizarre story, which is we bought a startup, which is a funny thing to do when you're a nonprofit. I love it. Yeah. Nonprofit and university. Exactly. There were two super talented engineers from Meta who had built a super little startup, which was a really good console that you can use for review and some enforcement, right? Because, for instance, say you want to detect if child sexual abuse material is uploaded on your services. If you do detect some, you actually have to be able to review to ensure that you know what is going on.

And so for that, you really want wellness features. And then if you do find that indeed this has happened, you have an obligation to report it, if you're in the U.S., to the National Center for Missing and Exploded Children, NECMEC. And that report is actually quite hard to do. So we bought this startup and we spent a lot of time working to open source the core technology and to really add on to it. It was quite lucky for us that some of the former customers like Notion decided that they were in it with us on the open source journey and really helped sort of like take that seed, that base and sort of grow it into what it is.

Camille FrançoisWe re-released Coop a few months ago as the world's first end-to-end tool that allows for the detection, review and enforcement of child sexual abuse material fully open source.

that's amazing because it means that now people who've always wanted to do this work can do it without having to knock on 99 million doors and pay lots of budget and what we're hearing is that it has sort of met its audience of again like people who were willing to do that and didn't have the tool so I can go into into many sort of stories for what this looks like in practice but And the point is, we often underestimate the tooling needs that are not just detection. On detection, there are still things that we are not very good at detecting. At ROOST, we have the ROOST model community. It's a really nice community where people come and actually share their safety models.

So we talked about OpenAI doing this with us a few weeks ago. Mistral said, hey, actually, we also have a model that we use for safety. We think it's quite cool because it's bring your own policy. It's multimodal. It's called Shell Stryle. We're bringing it to the ROOST model community. That means they have to do office hours and the community come. Then say, OK, I'm testing it in this way. I think this is working. I think this could be improved. Everybody shares the feedback. And then we sort of continue iterating, right? There's also areas where nobody has a good solution. And that's where we as ROOST invest in scientific partnerships with independent, rigorous sort of scientific and academic organizations where we say, OK, here we're going to need a big collective effort and push to actually bring some new forms of detection into the world. Can I just ask, you know, you said like, you know, thankfully Notion came on the journey and there are benefits. Like you said, Discord, you know, they did it, but then they get the best of what everyone is doing because everyone's building off them. Like that's obviously the whole ethos of open source. But are we like relying on the kindness

of strangers? I feel like people are so worried that we're like companies are never going to police themselves or they're not going to do it if. How do you make it so every company says, hey, I'm going to be a part of this ecosystem? Yeah, I want to figure out answer that question. You know, I think that. Yeah, I think that leading by example is cool. I do think that when we were early, the few companies who said, hey, we're just we're going to do that. We're going to we're going to invest in this. We're going to put our own resources. We're going to share our own systems. We're going to maintain the commons. I think they started something that continues to grow, right?

Now we have more and more contributions to ROOST. I want that sort of race to the top, right? I want people to look at the benefits that everybody gets by sharing this technology, the benefits that they get back in super talented engineering, the benefits that they get back in trust, right? And of course, I want more people everywhere across the industry to say, hey, actually, I also want to share my tools and I can see that there's something that's slightly better than what I have. So I'm going to adopt that. The other thing is we're very focused on who else should be part of this conversation.

I think Silicon Valley for the longest time has developed safeguards and safety systems a little bit in isolation. And when we think about, for instance, youth mental health, which is a big topic for us, there's a lot of really important expertise that hasn't been at the table. Working in the open allows us to work hand in hand with clinicians, with sometimes victims and survivors, right? And to bring a lot more in different perspectives than to just do this in isolation in a building in Mountain View or Menlo Park or whatnot, right? One of the things I think people easily understand or if they pay attention as to why a safety commons and open source is useful in a cost structure because something based off of in a quality structure because multiple people using it and fixing it is good.

Say a little bit about how it might also help startups succeed better. Yeah. Right, because part of the thing is like having them grow with it and have it as part of it is actually, in fact, something that isn't just a, oh, I'm checking the box to make sure I'm not doing something bad, but it's also helping me succeed. Yeah, I love this question. Let's take two different types of startups as a thought exercise. The first one is you're a small startup and you're going to build a social product and you want to focus on making that experience amazing. Maybe you're part of your social product is people can connect and maybe there's a chatbot, right?

What I see from the younger builders, right, the younger generation that's building products every day, is they have seen firsthand the harms of what it looks like when it can go wrong. They have sometimes personal experience and they want to build safety by design. They actually don't need to be convinced, right? And they think that safety can be an accelerant for them. So they think that, you know, if they can get this technology, then on top of it, they can build a better product. There's also markets in which this is going to be the cost of doing business, right? So, for instance, in certain regulations, you have to go and filter out child sexual abuse material from your platforms or your data set that are going to be used in AI, right?

And being able to access the technologies to do this in a way that you can trust, frankly, like, allows you to compete in the market, right? So it's a pro-competition thing generally, helps people build better products. What about safety startups? That's also really interesting because it's difficult to have a safety startup because, as we said, most of the companies that you're going to try to sell into or organization that you're going to try to sell into have to pick things up from scratch.

Camille FrançoisBut if everybody has a shared foundation of tools, then it's much easier for your clients to plug into the amazing technology that you're building.

So we see already at ROOST safety vendors engaging with our work. We have this amazing company called Zentropy who has been contributing fantastic open-weights safety models. You can bring your own policy and tune it. And then we have another amazing startup called Musubi who's actually selling a hosted version of Coop. right um i'm laughing because in my head i think about this this project as musou coupe because in french means flying saucer i'm reminded that this is not official marketing for it but the point is you have startups who are like okay this is really cool we can build amazing services and technology on top of the shared safety commons right so if we do this right the the shared safety commons is not just a hippie idea it's also something that's an accelerant to good business startup and innovation everywhere. So clearly it's important for safety for us to get it right.

So quality is really important. But you also did mention cost, that this is actually cheaper. And so one platform estimates switching to Coop cut its tooling costs by 95 percent. Yes. Can you dig in there? Like what made the old approach so expensive? You know, I don't know what they were using. But I think what matters is we are in an environment in which, sadly, people are, you know, reducing the budgets that are dedicated to safety. And so we want these resources to be focused on problems that are interesting and unique to those platforms, right? You don't need to use your resources to rebuild over and over basic foundations to build a safety infrastructure on.

And we want those resources to be focused on the great people who are going to do this work, right? On retaining the people who are going to think about the policies or are going to investigate the incidents. And so it's true that in this context, giving everybody a shared layer to start from doesn't mean that it's totally free in a sense that then you have to host it, right? You still have to invest. But it allows people to focus their budget where it will have the most real impact on protecting people on a day-to-day basis, which we're happy to enable that. So one of the things, you know, that folks get when they understand open source on the defense is lots of different people can look at it, can rattle it.

Improvements are shared across. And so there's a set of things that, you know, traditionally, cybersecurity, et cetera, you know, the open source protocols for the fundamentals of the Internet, they get better. But one of the things that also is a tradeoff is the attackers can then also look at it. Yeah. So say a little bit about the principles about this tradeoff between getting stronger defense by having a commons, but also how do you deal with the fact that the attackers can also see the commons and what that might mean for helping people navigate this? Yeah. Thanks, Scott. This is not a question that we have to take from scratch. It has been structuring the entire field of information security for over a decade. And so I think here we can take some inspiration, frankly, on how things work on the security side.

So if you are doing reverse engineering of malware, for instance, more often than not, you end up using open source tools to do the analysis. But because everybody uses the same tools, you can share the rule, right, amongst analysts.

Camille FrançoisAnd saying we open source the infrastructure doesn't mean saying you have to share all of the rules for all of the investigative work that you're doing.

But it does enable defenders to collaborate and then go faster together. And so we see this in Osprey, for instance, right? Saying we have a rules engine that everybody can run for free in a way that's permissionless and you can adapt it to your infrastructure doesn't mean you have to publish all the rules for how you define and catch spam and define and catch coordinated harassment. But if you do find something that you think is a particularly gnarly activity that unfolds on your platform and you have good reason to think that it's also affecting another organization, and if they're also using Osprey, you can share that rule and then you fast forward the collective defense.

And so it's always that sort of thinking and muscle to develop between what needs to be in the commons to really lift up and enable all the defenders and frankly, widen the circle of what is protected online and across AI surfaces versus what is sort of like organizations own things to keep and share. And, you know, different organizations will have different preferences on that, too, which is totally fine. Actually, I have two questions for you. One, coop, osprey, ROOST? What's the deal? What's the deal with birds? Yeah, we like a good pun. Eri, you would be surprised by how many chicken puns we're able to produce a minute.

So just random. No rhyme or reason. We just like birds. We just really like puns. Birds of a feather. Birds of a feather. Okay. Exactly. Fly together for safety. Exactly. um it's it's sometimes we also repeatedly use the same pun so now every time i say well you know it's a chicken and egg problem then i think yeah i will remind everybody that our discord where all those great puns are flying around is completely open and if you want to engage in technical safety discussions while making very fun puns uh if you want to be punny uh you can join us on the discord OK, rife with dad jokes. Great. Glad we solved that. Second, perhaps more importantly, the conversation is sort of premised, you were saying, of the problem with safety and safety startups is, you know, Smyte gets acquired and shut down.

And so there's a rug pull, people who are relying on this. But ROOST is a nonprofit. And so it is also predicated on the idea that people are going to fund it. So, one, how do you ensure that funding continues and what happens if ROOST's own funding dries up? I feel like you are inviting me to make a very public call here. People come and fund our work. Please come and fund this work. Come and fund this work because it has a very, very tangible impact. And fund this work, too, because it's a path to resilience. I love your question because we are building this Commons in a way that it can outlast the organization.

Yes. The idea of our work is to bring enough people on board, to bring enough of a community that innovation happens, more people have access to these tools, trust gets built. And if tomorrow we disappear, all of this stays there. That's the difference between an open source commons, which has this resilience, right? If tomorrow there's no worries, people can absolutely still download and maintain and handle Osprey and Coop. Now, there's a lot of work that we do to, again, like catalyze those commons, make sure there's a common roadmap. I'm particularly excited about some of the scientific efforts we're also building to because now that we have the commons roadmap, we know where there are real gaps. And we're able to join forces with institutions like Mila in Quebec or with the UMass Rescue Lab to say, OK, let's focus and sort of plug these holes and gaps.

But we are building with resilience in mind. I think that's extraordinarily important in the current times, right? Idea is really not to replicate a hyper-centralized system. That's why we chose the open source way. Makes sense. So one of the things I think the resilience that you've described is absolutely fundamental, one of the great things around open source, which is because it's a collective digital commons, it gets all the features of that, which is it can be, it never dies. It can be resumed. It can be steered in different directions. Multiple players can engage in it. It's part of the thing that is magical about open source.

But there's also a lot of different constituents. So there's nonprofits. There's universities. There's companies. There's engineers. There's random people. Say a little bit about kind of how these different elements come together because I think that's important for people understanding the resilience and the vivaciousness and the liveliness. And then we're going to start shifting into, and how does AI begin to change this? The magic of having an open source safety commons is that once the wheel starts turning, people start seeing how they can plug in. And so you have people who, for instance, don't have technical skills, but are really, really good at building policy.

And as new big labs are saying, oh, here's a safety model you can use, but you need to bring your policy, a few people are saying, oh, I can write a very detailed, clinically informed description of what suicidal ideation looks like in the context of an AI chatbot. And that happens to be super helpful for people who are going to use AI models for this. And I'm going to contribute this policy in open source in a policy pack. You, of course, have great engineers who are coming and are saying, I see this tool. I have an idea for how to improve it. I'm going to submit an improvement. You have scientists who are saying, oh, I've always wanted to go and actually test some of these tools.

And I'm going to go and I'm going to do this. And then I'm going to contribute a few insights from fields that have been traditionally left out of how we build safeguards. that actually could benefit from this. And so the work of an organization like ours is to architect and organize that big collective will around these tools is to make sure that everything is interoperable, right? Otherwise you have like an island of misfit toys with a bunch of things that don't work together, right? What we want is a good, strong interoperable commons. We have people using all of our tools together, right? So Matrix, for instance, is using both Coop and Osprey.

And then you have people who only use one part of your system, right? So, you know, the work is to, again, orchestrate and organize this big collective effort that brings a lot of very diverse stakeholders. But that diversity is the strength of an open source commons. And I will say in the team, we have amazing expertise from the sort of traditional open source side of things. So I have a fantastic CEO and who used to run open source at Google. And so she understands how you sort of bring all these pieces together. Cassidy is our absolutely fabulous community manager. is so good. It's sort of like, you know, seeing two people go in very different directions saying, actually, I know how we bring this into a community through office hours. We further the project. And so, yes, there is definitely a muscle to orchestrating that big collective and very diverse community around it. But this is what gives it the strength and this is what gives us tools that are actually robust and, you know, better than people doing this alone.

Before we continue down the AI thing, I think one of the things also may be helpful for a lot of people to hear is how does being open source help engage the kind of intense intellectual and scholarly work, research work at a university? Because that's something that most companies don't have any real access to. And that's one of the many reinforcement loops that actually provides an advantage here. Yeah, it's a language translation question, right? So if you take this work and publish it with the right colleagues and in the right journals, then, you know, people who come more from the scientific and scholarly practice see it and say, oh, I understand the language of working through a data set in the paper.

I can add my piece to this. And so, again, it's about sort of like translating that work across different audiences to engage everybody. Right now, our director of product, Juliette, is doing this on the difficult problem of grooming and thinking about, OK, what can we learn from law enforcement who's working on this? What can we learn from the different really talented academic teams that have been tackling this problem and from the many corporations in the ROOST community who say we have tackled this, we have some context, we have some data? How do we pull all of these pieces together that are currently quite separate to really radically accelerate innovation when it comes to grooming detection?

We've started this work. Roblox, for instance, shared a first packaging classifier that's focused on grooming detection, got feedback from the Ruse community. Both people who are trying to use it, people who are thinking about grooming, took that feedback and re-released something that's faster, easier to use and load better. So it's sort of doing this orchestration of, OK, let's go and grab these perspectives and let's organize some collective push towards very focused, discreet, well-described projects we can tackle. So coming into the AI age, AI is making it just easier and cheaper for folks to watch apps, chatbots, everything.

And sometimes it's small teams, sometimes it's one-person teams, and they could have no background in safety whatsoever. However, how can these small teams make things safe right from launch? I have a lot of thoughts on this. I have personal thoughts, professional thoughts. I suppose I should start by acknowledging that I am myself a compulsive vibe coder. Great. I've coded a number of apps, including like family apps. And I am excited by the fact that people are able to use AI to bring their own vision of something into life faster and faster. Our big vision at ROOST is that we need to bridge the gap between how easy it is to come up with a new app, a new service, and how hard it is to actually do some of the important steps of making it secure.

I think where we feel that the most right now is on child sexual exploitation. We have a lot of builders who are coming in and saying, we want to do this. We want to do this right. And we're very excited when that happens. We're like, OK, we will take you through those steps. But the reality, for instance, of removing child sexual abuse material from a foundational data set that's then going to be used to train an AI model is that right now it is a hero's journey.

Camille FrançoisAnd you have to go and then register with, you know, the authorities on one hand and then go ask for a hash bank and then go and implement, for instance, HMA to use this hash bank.

And then you have to go and ask for a novel CSTEM classifier and then implement it into something else. And so a lot of our focus is how can we make that process much easier? the overall vision is that one day it's going to be as easy to launch an amazing website and app than to just type ROOST at safety. Totally. And it will just do it. Or even, you know, by default, those tools will tell you, hey, so fun that you've added a field in your app that allows you to upload, say, kids' books into an app. Also, you may want to run a moderation layer here. And here's why and here's how. And we'll make it so easy for you.

So we want to make that much easier. There's a lot of work to do to do this. And we're excited to work again with others to go in and tackle that gap. So you started with a partnership with OpenAI and the GPT-OSS Safeguard. So what would encourage more frontier labs to contribute to ROOST? And are the various frontier labs also contributing a bunch of tokens relative to the vibe coding side, which is, you know, kind of the equivalent of like mythos analysis, other kinds of things? Because that would strike me as a great thing to be doing. I'm hopeful that there's a positive answer here, but I figured I'd ask.

Let's make it happen. I think we're also starting to see consumer demand and regulatory demand for that, which I'm excited about. and one way to think about it is that's kind of what we expect from other industries so for instance really if you wanted to buy a car and i was trying to sell you my car and i would tell you it has airbags i don't know if they work really well if you're short also it has a seat belt i think it's made of paper but i'm not sure but also i don't really know what other people use and you're going to have such a great time. You would be like, I want none of this, right?

Because in an industry like, you know, cars, we have a set of standards and shared references and shared technology for what we know keeps people safe. In, you know, most cars that you can buy these days will use a three-point seatbelt that was developed by Volvo. And in 1959, they said, we think it's a really good model. well, actually, we're going to give all the other manufacturers the right to do this for free. And then we're going to tell people how we do it so that we can continue improving on it. Right. And that's sort of our expectation for how things work. That's also what enables innovation. Right.

The reason why we have super fun cars that go really fast is because we know how to make them secure. And so I'm also hopeful that there's going to be increasing demand from consumers and regulators to sort of tell people like, hey, if you're going around town telling everybody that you feel super satisfied with the way you keep people safe online and on AI, maybe you should give everybody those tools. Maybe you can join me in having this public call to all of those amazing AI labs. I know they have more to contribute. And I can tell you, people are hungry for this tech. What's really fun when you have an open source stack is you discover people using your tools in production through different and funny ways.

So I'll tell you two stories. Can I do two stories? There was a small fandom app, two people. They found Coop through LinkedIn, actually. Promise. Yeah, great. And they installed it totally on their own.

Camille FrançoisWe didn't know. And they deployed it. And in two weeks, they found 19 incidents of child sexual exploitation that they were able to report to NECMEC.

And then they reached out saying, thank you for enabling this work. We really wanted to do that. We didn't have the tools to do it. Then we worked with NECMEC to say, OK, well, what does it look like to do this experience from the perspective of somebody who has a team of two? And I'm not saying an engineering team of two. I'm not saying a safety team of two. I'm saying a team of two people, everything two people. And to think about, okay, how can we make this as easy as possible for builders who want to do the right thing? That teaches us that the coalition of the willing of people who want to get this right is so much bigger than the people who are currently equipped with the tool to do this, Which, again, is good news in our ability to quickly make a dent, a real dent in protecting people online.

Other story, actually also LinkedIn related, quite fun. One day, the lead contributor on Coop, so the formidable engineer called Juan, receives a message from Atlassian saying, hey, we're going to submit a pull request. Can you review it? And we're like, Atlassian's using Coop? We didn't know that. That's cool. They're big. We're excited that large companies are staking, you know, some of their safety infrastructure on these tools because then it's exactly what's going on. Then they contribute to it. So it's absolutely awesome to discover the use of your tools by amazing engineering teams knocking on the door saying, hey, actually, yes, we've been using it. Actually, we think we can improve it.

But today, if I just think about Coop and Osprey, I'm not thinking about the ROOST model community and the amazing AI safety models, just Coop and Osprey. The people whom we know are using them in prod reaches around 370 million monthly active users across platforms and organization. That's our conservative count. I think we can do so much more. There are so many more organizations to protect. What's fun about this number is that it's both large platforms and small organizations, two people team. Or for the first time, we also had a journalism team say, hey, we're building our own community. We actually want it protected and we're going to use Coop.

That's awesome. Well, so we've talked a lot. There's a lot more to online safety than keeping kids safe. But we talk a lot about keeping kids safe. And Reid and I also talk a lot about AI companions. And so getting a little specific, when you're thinking about creating an AI companion for a 15-year-old, what would a genuinely good AI companion look like? And what are the open tools that the developers would need to build one? This is a hard question, and this is an urgent question. What we know from the usage data of AI companions and kids is that kids are increasingly turning to AI companions.

And when we look at the types of conversations they're having, they're also discussing feelings and their own problems and seeking support. And so there's a lot of really important and consequential decisions to make, science to develop, safeguards to build around what does it look like to keep those interactions safe. I am not part of those who think that we should just unplug older kids. I we talked about this I grew up playing with technology I think it gave me a lot of meaningful connections and friends and I think it gave me a meaningful career and we want kids to be able to play with technology and to engage we also want them to do so in a way that's safe I think safe in this context include things like products should be honest about being AI we haven't really across that branch just yet.

For instance, I think a lot about AI and toys. Today, if you're going to sell a toy that has AI embedded in it, you don't actually need to disclose that. And so you may end up with having an AI toy in your house without anybody sort of thinking twice about it. There, I think, consensus around companions respecting age-appropriate boundaries and about supporting real-world relationships, right? Ultimately, we don't want those companions to isolate young people from their support networks, their friends, their families and their community. I don't think any of that is solved. I think this is a very active area of focus for the field, for ROOST, of course.

And because this is not solved, because this is extraordinarily urgent, and because this is hard, that is why we're doing this work in the open with scientific institutions like Mila. Our suicide prevention safeguard, we released it in July. And what we're saying is we're going to do this work in public and incrementally so that everybody is able to come, join in and say, OK, let's get this right. And we can sort of divide and conquer together to make sure that, again, we accelerate the pace of innovation. Because, yeah, those usages are widespread. And I think we've heard firsthand and in extraordinarily dramatic ways the cost of getting this wrong.

So it sounds like you don't think we're quite there yet, that there is more sort of research and tooling needed before we're able to safely deploy. Yes. I don't think that we know enough to say we absolutely got this one solved. Yep. I am very grateful for the builders who are trying to get this right. I'm even more grateful for the builders and scientists who are trying to get this right in public and with us. But I do think there's a lot of work to do here. And, you know, again, I think that the usage data shows that this work is really urgent. One of the classic challenges is if someone's building something, trying to include kids, you kind of have two paths.

One is we're just not going to include kids or we're going to include kids and we're going to learn as we go. Probably we can only get success by doing the second path. But what would be some of the, oh, make sure you're doing this. Make sure you're engaging this. Make sure that you're asking these questions so that as you're learning, as you're going, you're contributing back to everyone else. And we're making the environment the right kind of safe environment for kids and elevating environment for kids that they absolutely deserve. Again, I think that there's sort of some of those big questions, right?

Does everybody know that there is AI involved, right? Do you have a good sense of what AI is involved? Which safeguards are running around it? Do you have protections around some areas like self-harm? Can you ensure that, again, you're not building a system that is going to lead to more isolation? These will also depend on the specific types and usage, right? You're going to have somewhat different conversations. Again, if you think about like AI embedded in toys, then if you think about multimodal AI, where you have a whole lot of different issues around images, for instance. Right. And similarly, building for a 12 year old is a very different story than building for a 17 year old.

So I think we need to sort of engage with all of that nuance and take it with utmost serious and do this work rigorously, scientifically in public and say, this is what we know. This is what we don't know. And this is the reality of how those tools are being used. And frankly, of how kids are being harmed by it today in those very specific ways. So if we brought in from just kids to sort of look at looking at the whole ecosystem and thinking about that chatbots, language models, agents, like they're delivering a lot of services that used to be delivered by humans, especially in the safety world.

So is there a specific safety capability that you're like, that would make the biggest difference if we could bring that online? And like, how would you show that that works in the real world? Yeah, I mean, I have a long list of CCL need. And again, I think there's a distinction between like problems that we kind of have solved for, but they need to be more robust, like better distributed, battle tested, invested in versus problems that are a little bit more new that we're trying to wrap our head around and thinking like, oh, that is really bad. How do we tackle this? Yeah. I'll give you an example of the latter.

Please. Non-consensual intimate imagery. Today, one in 25 American teenagers says that they have had a deepfake nude produced of them. I'm not saying one in 25 tell you, I know somebody to whom this has happened. One in 25 tell you, yes, this has happened to me. And this is an enormous crisis because, of course, those deep fakes are then used for extortion, for different types of humiliation, harassment and harm. And thinking systematically about, OK, how do we tackle that novel usage with the widespread scale of that problem and the very real consequences that requires new creative thinking? and frankly like a whole of industry and scientific approach on where do we focus technical defenses and make sure that they're distributed as widely as possible to get ahead of the fact that this is an enormous wave of harm coming for teenagers.

I wouldn't have even thought of that one. Now it's going to haunt my dreams. So thank you. Well, that's why we have all the puns is because let's get it solved. Let's get it solved. Yes. So, I mean, as part of that, So how are the collaborations with the Frontier Labs and other AI labs? What kinds of things need to improve? What are the ways that government and civil society can help this as well? Like how do we get that in a faster and stronger kind of thing? For example, solving the, you know, whatever. I don't know what the exact term for the sexual abuse from deep fakes is. But like solving that is critical and the kid doesn't even have to be a user of it for that.

Right. It's like that's the it's not even like, yes, how they use it. It's other people using us. So what's currently going on and what should we be doing to make it better? I think this is a call for everybody to roll up their slaves and work together. We need more attention to this problem. There's a lot of really good civil society groups who are doing these surveys. That's why we know that this is happening and who give us a lot of context on, okay, well, what does that look like for a kid? How does that happen? There are a lot of players who have technology to contribute that can help mitigate this harm at sort of the different areas where it needs to be stopped, right?

And we want that technology to be contributed in the open commons. We also do need a philanthropic organization to say, we're going to take a bet on the fact that we can accelerate this work. We can do this in public. We can do this with the rest of the scientific community. And as I said, I think that there are a lot of expertise that live in civil society and in the scientific community that for too long we've said, I know you don't belong in the safety and safeguard conversation. If we want your input, you will sign a stack of NDAs. You will come to some office in Silicon Valley. We will hear you and then you will leave and you will never know what we took from your remarks.

That's not a good model. That is not going to give us the pace and scale and excellence that we need in the face of these problems. So I think this is one where everybody can really bring something. I'm somewhat encouraged and also somewhat disturbed by all the conversations that we're having right now on pacing the frontier and how everybody needs to bend together to do something. There is also sub frontier problems that are very urgent, very catastrophic, and that we absolutely should bend together to focus and solve these problems. Right. And I mean, and so much of the conversation is also about global cooperation and what is happening around around the world.

And so you were recently speaking of the G7 in France and you were talking about the importance of open source, digital sovereignty, child safety. Can you talk about why this is so important in the context of European builders? And so, like, why is it helpful, especially in that context? Yes, I was invited to contribute to the G7 and not only because of some of my academic work with a broader coalition of colleagues on defining what we mean when we say open source AI. Because, of course, open source has had a traditional definition that we've worked through a lot for software. But when it comes to AI, it's a little bit different.

And in fact, there are different ways to do open source when it comes to AI. And open source AI can itself take different shapes. It can be just open weights, or it can be everything is open, including documentation and data sets and safeguards. Or it could be many different combinations of this is open, but not that. And you sort of have to think about the entire stack. And for each component of the stack, you have to think about what would it mean to open that? And then you think about, like, what am I trying to achieve and which shape of openness gets me there? That conversation is very relevant to sovereignty because, of course, you have the European Union, but also most recently Canada with their big AI strategy saying, well, we want to be able to own, not rent the different components of the stack that matter to us.

That is also extremely relevant for safeguards, right? Because similarly, if you are going to legislate on child protection, you also want to make sure that people, your own ecosystem of innovators have the means to comply with your law. Otherwise, the more we call for regulation, more rules on this topic without investing on open source tools, the more what you're actually doing is further concentrating power in the industry and closing down the market around a handful of incumbents who have the safety engineering tools and teams that are able to comply with those laws. So it's critical to your ability to protect your citizen and to make regulation in this area.

And it's also critical to your ability to have independent and strategic and sovereign versions of your stacks. Well, going a little bit more global, one of the challenges when we deal with a lot of the prospective harms is that, for example, abusive images can be created anywhere. and distribution networks and all the rest. So what needs to happen globally, right? Because one of the questions is there will be image generators, for example, running in the vast majority of countries. There's already open source versions of them or open weights versions of them, and there's nuances you were gesturing at there.

So what should be thought of as some good basics or some good starts for the G7 coming together, for the UN. We're here in UN General Assembly Week. What would be the kinds of things that would be good things to do globally? I think my first ask is never use the idea that you can't root out all of the problem to justify not taking the extraordinarily important incremental steps that really are going to save lives and reduce harm. Yes, there will be people doing this with models that we don't have control of in places that we won't see. But also, when you look at the way the majority of these harms are being perpetrated, it is happening with apps that are being advertised on social media, distributed through app stores, and in contexts where there's actually so many different levers that we can identify and say, here, there's a good chunk of the harm.

that if we focus and join forces, we can remove. So I'll say that's my sort of first, most important goal, which is like, let's get progress done. Then we'll worry about the sort of remaining part that we can't get to, but there's so many lives we can save, harm we can remove, good we can do by focusing on all these incremental steps that we're right now not investing enough on. You've described your generation as internet optimists who believe cyberspace would spread democracy and expose corruption by default. We all heard this. We were so excited. And some of it came true. There have been, you know, many good things that technology, social media, the Internet, etc. have brought about.

We didn't, though, predict all the harms. And, you know, we watched some of these beliefs get weaponized. And so is the optimism you still carry, is that a conclusion from the evidence that you've seen? or you are just an optimistic person? Like, how would you say your optimism has sort of waned and waxed over the years? There is somebody somewhere who could probably answer the question of what is optimism? Where does it really come from? And I would like to know. I think maybe in my case, perhaps it's a personality trait. I am part of this generation who is extraordinarily excited about the power of technology and internet. I remain extraordinarily excited about it. Over the years, I've had to work on violent extremism, terrorist use of these systems, foreign interference, hate mobs, suicidal ideation, child sexual exploitation. And so my colleagues often say that I am maybe the most optimistic person looking at the darkest corners of our technological life. I've seen the worst.

But I always see those reasons for optimism. The first one is that there is so much we can do. If we join forces, we know what to do. We just need to focus and we are still in an environment where we can make very tractable progress by doing things that we know how to do. And so that gives me a lot of optimism because then I know where to focus my action. Right. There's this question of like, does action come first or does hope come first? I actually think that hope comes from the action. The other thing is I am very excited by the next generation of builders. sometimes when we think about child safety we can describe the next generation as sort of a big unified mass of youngsters to which tech is happening but that's not the case there are also the builders of tomorrow they're also very creative and fun and there are a bunch of maker pirates within them too and it's fun to see what they do there's a hashtag online that's a cyber deck for instance, where you can see young girls take polypockets and inside they build mini computers and then they run local AI models. That's really cool. The next generation is building systems that they want. They're going to build a technological environment that they want to see. And I think we

need to bring them at the table, not just as victims, but also as builders and shapers of our technological futures and everything that we do at ROOST, we also do for them. And it's super optimism producing to see them knock at the door and say, hey, I'm going to build this weird app. I need all these safety tools to do it. And we're like, yeah, go, go, go. You don't need our permission. Go to your app. So yeah, that's exciting. And that produces optimism. So one of the kind of phrases around this that I really like is cyber peace. And it describes not just fewer bad outcomes, but actually a kind of a healthy and dynamic and growing system, kind of an equilibrium. And you've talked about it some today with pluralist, audible safety, you know, kind of ecosystems. Communities get more nuanced and detailed and specific to their community kind of agency. What do you think are the most important things to help continue, accelerate to that path? Like what should people understand about what kind of ecosystem we're trying to get to? Why should it be kind of almost like a, what, you're not using these tools, you're not participating in this, that why not, right? You know, it's kind of as as opposed to the why, the why not. What do you think it takes to get there?

So I've done my academic research and ultimately my dissertation on this concept of cyber peace. And that is because I was amongst a group who was studying cyber warfare. I had two absolutely fantastic advisors who encouraged me and supported me to sort of look at the problem maybe from a different angle. And I was taken by the fact that everybody was very focused on defining cyber war, defining the threshold of armed conflict. And it's, as I said, not unlike the conversations we're having right now with a hyper focus on the frontier. But there is all this great theory of positive peace building. What happens below the threshold? If we're so intent on defining cyber war, do we know how to define cyber peace? Do we know how to invest in the mechanisms that make our online lives livable, that make room for thriving, that make room for cooperation?

I think that the disappointing answer to your question is that some of that is the power of our focus. We get to decide where we want to spend our energy, where we want to spend our efforts, where we want to build, where we want to spend. And saying we care about making the sub-frontier, sub-war, space livable, we care about tackling these harms that are immediately in front of us is a decision we can make with our focus. and making sure that we can keep centering these in the conversations we're having around AI, around harms, around AI safety, I think is part of the problem. We don't need to have this opposition between catastrophic risk and immediate risk.

I don't think that opposition is particularly, of course, I understand why it exists, but we don't need that strong of opposition. we can make sure that we focus on both sides of the line and that we do both at once. This is just reminding me. It's like government works well when you forget that it exists. Like none of us know that we're getting tons of spam text messages a day because that system actually just works. And some of the spam texts are just filtered out. We still get some, but most are filtered out. Like all of these systems, when they just work, you in a good way get to take them for granite. It's sort of like the Maslow's hierarchy. It's like if safety is the floor, that's right. Like we have to have it exist before we can get to flourishing and before all these great things can happen. And so what does that actually look like once we have safety as the floor? Yes. And then what does that look like when cyber peace is real? Yeah. For a teenager, a platform, government, like how to paint that picture? I love that description. Like, yes, we want to become ubiquitous and you don't think about it. Totally. And things are just more safe. And that is just the way it happens to be. And frankly, we have that on some corners

of our online lives. How often do you think about the amazing work that Less and Crypt is doing? I don't know. Maybe you do. But generally, when people start a new website, they're not going to their office thinking, oh, I need to figure out cryptography, right? Like, no, we don't, right? We've managed to sort of like have building blocks and institutions that raise the floor for everybody. I think we can do this for safety. I think it will look different in different products because again, we don't know what people are going to build. What we know is there are fundamental things that need to be done in order for those users to be safe. That some of this we know how to tackle and we need to make those technologies available to everybody. Some of this we don't know how to tackle yet. We need to focus, make that scientific effort, build those new safeguards.

And then again, will this look like new, very fun chatbot interactions? Maybe. Will this look like less spam, hopefully? Less fraud? Please, that would be great. Less child sexual exploitation? Absolutely. Less interactions with chatbot that leads to violent extremism? Yes, please. All those things, right? But like, yes, the idea is exactly what you said, which is let's make it ubiquitous so that it's much easier to build from a shared safe foundation. All right. So let's go to rapid fire. And you can answer at any length. These are just the same questions we ask all of our delightful guests. So is there a movie, song, or book that fills you with optimism for the future?

Many. I will pick one because I think that's the game. Yes. So if you want to do two, it's fine. I am a Trekkie. I love Star Trek. I am a Trekkie of the Deep Space Nine category. And I'm not sure why, but it actually turns out that a few people around ROOST and in the ROOST community also are Trekkies. So I think that there is something about Trek inciting optimism about the future, optimism about our ability to collectively govern it at interplanetary scale. But yes, when I think about our board, for instance, we have amazing people like Audrey Tang or Megan Smith or Maria Ressa and all those people are also Trekkies.

And so it has happened that we have very serious ruse discussions and then the end is like, OK, OK, good. Live long and prosper. Awesome. What is a question that you wish people would ask you more often? I wish people would ask who should be at the table when we make safeguards. Again, I think we've inherited bad habits of doing this a little bit on our own, in the dark, in a way that's very opaque and sort of siloed. And there is another way to build in thinking about, well, who else has an interesting expertise to contribute on a very difficult societal problem is a question I wish people would ask more.

And I think we will bring more people and, yeah, more perspectives. So where do you see progress or momentum outside of, call it, you know, all the things we've been talking about, your industry, but like another area that you would like to shine a light on that inspires you? Well, it's interesting. I know you said another area, but to people who've been working on child safety, we are going through a very interesting moment where that conversation is becoming a kitchen table conversation. And so I am quite inspired and energized by the fact that many people around us are starting to think about, OK, well, what type of relationship do I want between young people in tech and what is my piece of it?

I'll give you a very bizarre example. I recently became a mother. And as a result, my daughter goes to daycare. And they had a conversation on, well, we have Amazon Alexa in the classroom. But the wake word that we use is Alexa. And in an environment in which this generation is going to grow up with companions, is that what we want? Because that's anthropomorphizing the relationship to the computer. Now it turns out, here's a Trekkie secret, that one of the other few available default that you can use for the wake word is computer, like in Star Trek. I didn't know that. That's cool. You can go and change it tonight. And so they were thinking about how to do their part in finding the right relationship between youngsters and machines as everybody sort of grows up against a rounded by AI and changed the wake word from Alexa to the very Trekkie computer.

And I'm excited by the fact that many people are having this conversation at so many different levels, right, in classrooms, in boardrooms. And this is becoming much more of a mainstream conversation. Awesome. In my house, my kids just try to say, hey, Google, and the craziest accents at the loudest volumes at the strangest tenors. So we haven't quite. No, that's great. We're not at that stage yet. But it's super. I think you have red teamers on your hands. I think you should send them to Bruce to go and red team some safeguards. They're very into it. All right. Our final question. So can you leave us with a final thought on what you think is possible to achieve if everything breaks humanity's way in the next 15 years?

And what's the first step to get there? Ubiquitous safety for everybody. Come roll up your sleeve and build it with us. We'll get there one step at a time together. Hear, hear. Hear, hear. Karine, thank you so much. Me so. Live long and prosper. Beautiful. Possible is produced by Pallet Media. It's hosted by Ari Finger and me, Reid Hoffman. Our showrunner is Sean Young. Possible is produced by Tanasi Delos, Katie Sanders, Spencer Strasmoor, Imo Zhu, Amon Suri, Danny Garrison, Trent Barbosa, and Tafadzwa Nima Rundwe. Special thanks to Surya Yalamanchili, Sayida Sapieva, Ian Alice, Greg Beato, Parth Patil, and Ben Rallis.

And a big thanks to Jeremy Ponack, Anne Bertuccio de Hollenbeck, Jacqueline Ortiz-Ramsey, Danielle Hogan, and the team at Lighthouse Brooklyn.