Representative panels need sampling and consent safeguards

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 22:54

   I don't understand polling. I don't understand when people say it's 53% of Americans think X or Y. It's like, who are these Americans? Who? I don't know. I've never met someone who's been pulled before or surveyed have you well you have so but yeah so like who are these people yeah so the way this operates um so these are people like just everyday people um who answers to who chooses obviously to answer to polling or decide to participate in these data collection processes the magic to this process often is however is knowing how to sample them So, social scientists have been developing methods for randomized samples, probability sampling to really get a broad understanding of the representative set of people.

2. 23:47

   In some cases, some of these panel companies, they specialize in this exercise. And what that means for them is they actually try to form a real relationship with people across the entire country. One of the strategic partnerships that Simile has actually is with Gallup. Gallup, of course, one of the most, I would consider Gallup to be one of the most golden standard polling company or company that is generating these kind of insights.

3. 24:12

   They have a Gallup panel of hundreds of thousands of people with whom they've actually built up relationship over the past decade, two decades, and so forth. But wouldn't those people be biased at some point or feel entitled to give them what they think would be like the best information based on their own circumstances or something like how do you actually know that that's valid information? Yeah so that we control for these a couple of ways we control one obviously is and the biases actually can clip up in different ways so there are different ways to fight for different kind of biases and you know not surprisingly this entire thing is a area of study like it's an entire academic field with many researchers working on this but for instance one of the clear bias that we often see is fatigue if you get asked the same

4. 24:59

   questions or even different questions but if you get pulled every single week they start to really get fatigued and they start to answer in ways or they try to gain the system oh i want to if i participate in this i get paid more so i just give some answer and i just participate you start to see those behaviors. So what you do is you actually time it so that the participation is far as few in between and sparse enough while being able to still maintain that relationship. And we would actually run studies that basically shows we have a new population versus population who had answered X number of questions over the past. What is the right amount of distance to have between asking different questions to get the high quality, what we would consider to the gold standard quality responses, we would actually measure that. Another thing that we would do is oftentimes in these kind

5. 25:53

   of studies, we also do attention checks. So these are basically questions that are dummy questions that are not actually a part of the core measurement, but you're basically sneaking in questions that would filter out participants who are not actually paying attention. What would one of those questions be? Question might be something like, imagine right after our interview you asked me what was the first conversation, what was the first question that you asked me in today's conversation? If I can't answer, that's tricky. You asked me about Elon, are we living in a simulation question? I think I should know that if I was actually paying attention to our conversation. But that's an example of attention check. I was thinking like, something's really silly. Checking for someone's attention, like how tall is a giraffe?

6. 26:45

   How many hamburgers can a human eat? So technically, that's actually also good attention checks. Really? You can use those if you want. Thank you. These are actually questions that we would sometimes ask too. Some surveys or data collection process will literally ask people to do very simple arithmetics. So that if they say 10 plus 10 is 30, then we know they're likely not paying attention. or they're not or or they are or they're they might not be bad uh good at math in those instances that brings up another question i don't know how to i'm just i'm now in the mood but are there certain populations you have to avoid like drug addicts mentally ill people like people that might skew the data in one way or another or is that part of the overall graph i know that's like an inappropriate question but I'm curious like what are the no-goes in who you survey and how

7. 27:48

   do you profile that out? So for us the no-goes are actually participants who cannot give informed consent. So we take informed consent very seriously that every member of panel who joined our study and our data collection process, they know why the data is getting collected, at least broadly, and they consent to that process. And they also have the opportunity to withdraw their consent. So that's something that we take seriously. So for instance, underage population cannot participate. So that's actually one blind spot that we own, that we do not collect data from underage population. And this would also generalize to other populations that cannot give informed consent.

8. 28:30

   Now, that said, aside from that population, where simile is different is we are trying to create model that can truly be a representative snapshot of who we are as a society. So there's actually no population that we should avoid. We actually want to know every single population that's out there. And this is also where, for instance, we're also quite different from simply creating a model, like language model or other modeling companies other modeling companies what they are trying to create is amazing reasoning machine that's great at coding mathematics and fundamentally trying to be good chatbot good assistant for us we are trying to create models that can actually model the diverse viewpoints the values preferences and taste of people a part of it is also then learning the good behaviors but also bad behaviors of people simile needs to be able to capture those