Mercor tunes candidate evaluations with customer outcomes

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 5:19

   Maybe for our listeners, too, on the Mercore side, you guys obviously have a bunch of uses of AI and screening candidates going through resumes. Can you talk through some of the different use cases that you have for AI and then what the stack looks like that you guys are building on today? Yeah, I think a good heuristic is just thinking about all the things that humans would do manually, creating evals over those and seeing how we can automate them.

2. 5:41

   So similar to how a human would review a resume, conduct an interview, and then rank people or decide who should be hired, we automate all of those processes. with LLMs. And so we have evals for how accurately are we parsing the resume? How accurately are we scoring different parts of the resume? How accurately are we asking questions in an interview, evaluating that interview, and then passing that all into model context along with the references or every other kind of data that we have on a candidate to make the end prediction around how well they'll perform. Is it mostly off-the-shelf models and you're kind of curating the evaluation and context around them? Yeah, there's a lot of off-the-shelf models for more basic things, but particularly for the hardest problem of making the end evaluation of a

3. 6:24

   candidate is where the post-training comes in and learning from all of the data we get from our customers of who's doing well, for what reasons, how can we learn from those signals to make better predictions around who we should be hiring in the future. Have you learned anything surprising about those signals of something that the AI found where you thought, you know, maybe this isn't how I would have thought about it or how humans would have thought about it?

4. 6:45

   Yeah, there's all sorts of things. I think that one of the key benefits of AI is that it's able to just go way more in depth about everything about a candidate and it's able to pick up on all the small details that humans sometimes miss or the vibe check sort of skips over because people already have their mind made up on a candidate. And so there's all sorts of like little resume signals if people have demonstrated extreme interest in a particular area where they're just doing it for fun, as you would anticipate, all the way to like different signals of whether someone studied abroad in a country that is where they're doing the end job, they might communicate better and be more conducive in a work environment.

5. 7:31

   And so there's lots of those little things that come up and are very specific to projects and customers. Are there certain things that you see kind of will always be done by people?