Randomized tests show which levers can shift behavior

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 42:29

   detailed psychological analysis. And Alexandr Wang was shocked by it. Is that an angle as well? Like how do you see how much of the data you need? So it is both how much, but it's about more about what data you need. So it turns out asking right questions actually matters quite a bit. So we do interviews with people just like what we're doing right now. In fact, where I actually got the idea of doing interview with people to learn about them was back in 2023 and 2024, I actually was on a lot of podcasts for the academic work that we led at Stanford. And I realized that after talking to these interviewers for about an hour or two, they know a lot about you, especially the good ones, if they ask the right questions, they know a tremendous amount about both your field and who you are. So that's where actually we got the inspiration from. So this is where we

2. 43:25

   ask questions that are both topical, but also quite general. And we found that asking people to just tell us the story of their life actually matters a lot. It reveals a lot of interesting pockets of knowledge about those people. But we also collect a lot of behavior data. Behavioral data might be observational in part. It's like transaction data, mobility data, makes a lot of sense. But what really matters is actually the causal mechanism. It's the randomized control trials. Imagine you have the same experimental setup.

3. 44:00

   and you're varying a particular variable whether it's a product feature whether it's a messaging and actually observing people's behavior change that data actually matters a tremendous amount because what we've learned over time is people are actually not that interested in predicting the future which is to me this was a little bit surprising but unless you're trying to win against a stock market what are you going to do if I told you that your market is going to crash in two quarters. They're all going to then ask, wow, that's shitty. What do we need to do now to change that? But it's always the second part that matters more. So what do we need to do now to shape the future? And this is where for us, it's not just about finding correlation. It's about understanding the causal mechanism, the causation, because that's ultimately the lever that we have to shift

4. 44:52

   the direction of the future. So who makes up the team? What kind of talent are you looking for? Is it researchers? Is it scientists? Is it engineers? Who's, how is it composed?