Higher classifier thresholds trade removals for fewer false positives

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 29:53

   have in our moderation system is this precision issue that you're talking about. And that is actually, of all the things that we announced this week, in terms of how we're going to update the content policies, changing the content filters to have to require higher confidence and precision is actually going to be the thing that reduces the vast majority of the censorship mistakes that we make right the um you know the the removing the fact checkers and replacing them with community notes i think it's a good step forward like a very small percent of content is fact checked in in the first place so it's is that going to make the hugest difference i'm not sure um i think it'll be a positive step though um and we we like opened up some content policies so some stuff that was restricted before we opened up okay that's good it'll mean that some set of

2. 30:39

   things that might have been censored before or not. But by far the biggest set of issues we have, and you and I have talked about a bunch of issues like this over the years, is like, it's just, okay, you have some classifier that's trying to find, say, like drug content, right? People decide, okay, it's like, the opioid epidemic is a big deal. We need to do a better job of cracking down on drugs and drug sales, right? I don't want people dealing drugs on our networks. So we build a bunch of systems that basically go out and try to automate finding people who are dealing drugs. And then you basically have this question, which is how precise do you want to set the classifier? So do you want to make it so that the system needs to be 99% sure that someone is dealing drugs before taking them down? Do you want it to be 90%

3. 31:28

   confident, 80% confident? And then those correspond to amounts of, I guess the statistics term would be recall. What percent of the bad stuff are you finding? So if you require 99% confidence, then maybe you only actually end up taking down 20% of the bad content. Whereas if you reduce it and you say, okay, we're only going to require 90% confidence. Now maybe you can take down 60% of the bad content. But let's say you say, no, we really needed to find everyone who is doing this bad thing, and it doesn't need to be as severe as dealing drugs.

4. 32:04

   It could just be, I mean, it could be any kind of content of, any kind of category of harmful content. You start getting to some of these classifiers might have 80, 85% precision in order to get 90% of the bad stuff down. But the problem is if you're at, you know, 90% precision, that means one out of 10 things that the classifier takes down is not actually problematic. and if you filter if you if you kind of multiply that across the billions of people use our services every day that is millions and millions of posts that are basically being taken down that are innocent and and upon review we're going to look at and be like this is ridiculous that this thing got taken down which i mean i think you've had that experience and we've talked about this for for a bunch of stuff over time and um but it really just comes down to this question of

5. 32:55

   where do you want to set the classifiers? So one of the things that we're going to do is basically set them to require more confidence, which is this trade-off. It's going to mean that we will maybe take down a smaller amount of the harmful content, but it will also mean that we'll dramatically reduce the amount of people whose accounts we're taking off for a mistake, which is just a terrible experience, right? It's like, okay, you're going about your day, and then one day you wake up and you're like oh my whatsapp account just got like deactivated because it's connected to a facebook account and the facebook account um is uh is is or like i'm using on the same phone as a facebook account where uh like we made some enforcement mistake and thought you were doing something bad that you weren't because our classifiers were set to too low of precision

6. 33:44

   has that happened before oh yeah where their whatsapp app got canceled as well yeah because So if your Facebook app gets taken out, like I say, if you have a Facebook and you have like a sock puppet account and the sock puppet account, you post offensive memes and you're generally gross. Yeah. If you get caught for that, does your WhatsApp get killed? Not for memes, but go back to like a very severe thing.

7. 34:08

   Like let's say someone is- Terrorists. Let's say the most severe. Sure. Yeah. Let's say someone is like terrorist content. They're planning some attack. Right. So we take down their account. Right. But then let's say that person can just go then sign up with another account. How does WhatsApp get connected to that, though? Oh, well, we run these different services, and if they're on the same phone, it's basically one thing that it's basically regulators or governments will come to us and say, okay, you're clearly not doing enough if you kick someone off for terrorism, and then they can just sign up for another account on the phone.

8. 34:43

   Okay. They also think, okay, well, we're not doing enough if we deactivate their Facebook account because they're like planning a terrorist attack, but we let them use all our other services. Right. If you're aware. Yeah. So if our systems think that someone is a terrorist, then you probably need to deactivate their access to all the different accounts. Yeah. They can't get on threads. Threads, Instagram.

9. 35:06

   Yeah. That makes sense. So it's you can understand how you get there. But then you just get to this question around the precision and the confidence level. And then you're just making all these mistakes at scale. And it's just unacceptable. But I think it's a very hard calculation of like, where do you want to be? Because on the one hand, like, I get it why people kind of come to us and they're like, no, you need to do a better job finding more of the terrorism or the drugs and all this stuff.

10. 35:37

    But over time, the technology will get better and it'll get more precise. But at any given point in time, that's the choice that we have to make is do we want to make more mistakes erring on the side of just like blowing away innocent people's accounts? Right. Or do we want to get a somewhat higher percent of the bad stuff off? And I think that there's just some balance that you need to strike on this.

11. 35:59

    We were having a conversation.