A large data-model migration can be worth reversing

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 35:36

   staff or IC7 was public groups on Facebook. So I'm curious, like the story behind your involvement in that and, you know, anything interesting that happened at that point. Yeah, so public groups was one of these projects that came out of this, this coping for, you know, like making Facebook groups more about communities. There's this like one very narrow change that we wanted to make that seems so simple on the surface, but it was so complex under it.

2. 36:00

   It's just funny, like explaining this to anyone that wasn't there, they're like, wait, this is like a one line change. And I'm like, no, it's not. It's like, it was very difficult to pull it off. And so the change was, in order to participate in a public Facebook group, you no longer have to join first. So you're saying you can just view, like you have read access for all groups, essentially, or public groups.

3. 36:21

   Read access for all groups, and for some groups, even comment access. So you can comment without joining first. Interesting. And this is a thing, you know, it feels like a one-line change. And it actually was a one-line change. But there's all these downstream implications that were so tricky. So one is, you know, in the data model, there's essentially a field in the database that was like group member.

4. 36:41

   And we had this like really intense technical debate about like these people that are commenting in a group, are they group members? And the model also changed where before to join a Facebook group, an admin had to approve you. So there's kind of a vote of confidence that you can be in this group. And then after we switched to this model where to join a public Facebook group, you just essentially press like follow.

5. 37:04

   And we actually went back and forth. Should it be join or follow? Like what's the right verb to describe this? But it was essentially follow because there's no reciprocal action. You know, if you follow a group, are you a member? Like, should you be stored in that same part of the database? And we just went back and forth on this for a while. And I remember at the time there was this like really senior engineer, Bob, he was kind of the most senior engineer in the in the org at the time and he felt very strongly that it should not be the same thing and he kind of pushed us pretty hard um even though it would be a ton of engineering work to migrate stuff to make it a different thing and so we did this work um because he was actually one of the early engineers on facebook group so he knew it really well

6. 37:42

   and he felt pretty strongly there's a bunch of these other like downstream changes around uh like moderation and different new like admin tooling that admins would need to handle kind of the influx of spam and things like this and i remember at the time thinking like if anyone can make a comment the comments are just gonna be like filled up with spam and i had to hard i had a hard time kind of convincing people of this and so at some point i built this like monte carlo like visualization of how this would work and it was just like this like really simple kind of like scratch pad of you know like a comment comes in there's a certain probability of it being good or bad and then like what actually happens to comments and i think that actually did a pretty good job of convincing the integrity teams to jump in and help with this. And so at the time,

7. 38:22

   the pages integrity team jumped in and they helped with a comment ranking because kind of ranking spam comments lower was the main technical mechanism to make it so people don't see these comments. So there's a bunch of these like pretty gnarly downstream implications of letting people participate. There's also this data model migration that we're doing. And so to do all this, we had to staff a big team to kind of make this happen. And so we hired a new director, Yamin, who hired a bunch of engineers there's a bunch of internal transfers so some of the most senior engineers from the org like uh there was like henry henry long uh joe chetham there's like a few other engineers and they were all working on this and i was the same level with them i was like a you know an ic6 at the time and so were they and i remember just feeling this kind of imposter

8. 39:08

   syndrome of having to kind of direct them and kind of point them at work knowing kind of in my mind that were the same level even though levels are hidden you kind of you know you know through like rumors and stuff who's what you know in hindsight i think this is sort of like misplaced imposter syndrome because levels don't matter at all this is my current view and you know some people that are very junior can shoot way higher than that and just give you amazing results some people that are very senior can give you terrible results and so the level actually doesn't matter that much but at the time i remember just like really thinking about this and it it was just kind of hard to step into this role and eventually i did it and it's funny eventually the thing that got me the promo to ic7 was reversing this decision that bob did because he wanted to do this big migration

9. 39:58

   and we did it and it was just Like, dude, it was so much work. It was like six months or a year of work or something, just migrating just hundreds and hundreds and hundreds of call sites to do this correctly. And then technically, I felt like actually what we did is we essentially just added an if-else at every single one of these call sites. In the process, we audited all the call sites. We kind of knew that it was safe, but we didn't actually change the logic.

10. 40:22

    And so actually what we've learned is that, yes, member is the right field to model both followers and group members. This was the right decision. And so I pushed the same engineer that did this to then undo it. It was the right thing to push this engineer because it showed maturity on his part that he said yes and was able to do it. He also had the most context technically so he could do the best.

11. 40:42

    And I think for Bob, he felt better about me as a technical leader because he knew that I was willing to push back on decisions that even senior folks make. And in the end, this was the right thing. So we reversed the migration. It also took a long time to do it. But in the end, it made it so everyone building on this infra could do it. And everyone wasn't always constantly bumping into this, like, should I use this field or this field?