Knowledge workers may shift toward creating evals

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 26:50

   As you think like two years from now, like where do you think this is all going? And like, do you think this is actually a part of your business Or in two years, is it like only the expert of the experts that are required? I think it's a huge part. And the reason is like I mentioned in the beginning, we started the business because of this notion of labor aggregation and that it feels like the way labor is allocated and the economy is wildly inefficient and we could make that much more efficient.

2. 27:13

   But a big part of that is making a bet on like what will humans be doing in the economy in five years? Please tell us. Which is a huge question for everyone. At least everything I'm seeing is leading me to believe that it's far more structurally efficient for humans to create evals over the things they don't yet know how to do or models aren't able to yet do than it is for them to redundantly do that task all the time.

3. 27:35

   And so I actually think it's highly probable that a huge chunk of knowledge work just trends towards creating evals. And it might not be the rigid context that we have right now of people working on an annotation tooling. It might be much more dynamic and then talking to an interviewer about how to solve their problem. But I think that that is going to be just like a huge part of the economy. And it's one thing that I think very few people are aware of yet because so many of them conflate it with what's happening in the SFT and RHF market, where a lot of those data types just aren't as useful.

4. 28:11

   as they previously were and budgets for them are coming down. What do you think will be the most interesting skills for people to develop or kind of, I don't know, if you were to advise someone that was in school, you know, what to study or focus on, where would you steer them? I would definitely optimize for like a fast rate of learning because things are changing so quickly, right? It's hard to know.

5. 28:32

   Like there's so many of these things that people didn't think the models would be good at for a long, long time that they just got really good at really fast. I would say like work with AI as much as possible. One thing I hear from people in our marketplace is that they love the fact that they just get to play around with these models all day. Like they get to think about like, you know, they get to spend hours thinking about a problem that the model's not going to work at, not going to be able to do.

6. 28:54

   And like, what are the things the model is like missing out on? And they say that they build a lot of valuable skills that help them to know in their workflow as a McKinsey analyst, where should they be using AI? Where should they not be using AI, et cetera? And so I think just spending as much time with the models as possible and getting very familiar with the things that they're good at or bad in a particular domain is really helpful.

7. 29:16

   But it's hard to say, like, be a software engineer to be a software engineer. Yeah, it's interesting that, like, you know, obviously, like your point that so many, many more of us will be spending time, like, training these models. And, like, you know, there's almost an infinite amount of things. Obviously, there's, like, hard skills that have, like, right or wrong answers. But then there's so many, like, just subjective things.

8. 29:34

   And like, maybe in the future, I don't know, we get paid to just train our own individual models for us. Like, totally, totally. Yeah, I think that'll be a big part of it. I would say one other thing is that people should focus on domains where demand is very elastic. And so an example is, I think there's demand to build 100 or 1,000 times more software in the economy, right? And maybe it's not like 1,000 times as many web apps, but it's like more feature iteration on existing products, better ranking algorithms, whatever it is.

9. 30:01

   versus other roles where demand is probably more fixed. Like we only need so many accountants, right? And like so much of an accounting function. And so as much as we can focus on those things that there will be vastly more demand for when we're able to increase total productivity is probably also a safe bet. Yeah, that's a great way to put it. I had a founder I was talking to the other day and he was like, for all this talk about software engineering going away, I really could use a lot more software engineers.

10. 30:32

    I know. It's something I'm really excited about. Like if they made our software engineers 10 times more productive, we'd probably hire more software engineers, right? And so I think that there's always like interesting curves around demand and how pricing will implicate it over time. I mean, obviously I imagine when you started,