The alpha, twice a week
That was 6 min.
Get the best new moments in your inbox every Wednesday and Saturday. It's free.
Check your inbox
We sent a link to your address. Click it to finish signing up. Not there in a minute? Look in spam.
or
Latent Space · 6 min · Nov 9, 2026
Building a training task starts with the whole work environment
Garrett Lord walks through building simulated tools, realistic files, prompts and rubrics, then testing rollouts for failures.
In this episode
Full analysis- 1:25:35One goodwill error can invalidate an otherwise polished deal model5 min
- 1:46:36An agent can beat an Excel grader by listing every number2 min
- 2:11:20Loan automation can produce illegal pricing from benign inputs5 min
- 5:12Building a training task starts with the whole work environment6 min
- 15:57Labs test data by training models, not just inspecting examples6 min
- 1:31:00Expert rubrics use gates for errors that poison later calculations5 min
- 1:56:45A safe redirect can still be wrong after a child-harm request4 min
- 2:20:01As agents gain autonomy, alignment must enter the data5 min
“End to end, these things can take, I mean, they take days. Um, and that's for one unit of data.”
Garrett Lordat 10:58 · Latent SpaceTakeaways
- Simulated tools need realistic, internally consistent files.
- Experts review rollouts for reward hacking before tasking.
- A single unit of data can take days to produce.
About this segment
Garrett Lord walks through building simulated tools, realistic files, prompts and rubrics, then testing rollouts for failures. A single task can take days, so the work is environment design and expert QA, not simple labeling.
Co-founder & CEO, Handshake
Built Handshake’s career network and launched its AI business supplying expert data and RL environments to frontier labs.

Shawn Wang (swyx), Host · Co-founder, AI Engineer
Wrote “The Rise of the AI Engineer” and founded Smol AI, maker of AI News.

Alessio Fanelli, Host · Founder, Kernel Labs
Partner at Decibel, backing technical founders who build developer tools.
From “What Frontier Labs Buy: A Data Deep Dive with Handshake AI — Garrett Lord & Team”, Nov 9, 2026. Watch the full episode
Thanks. We'll look at and fix what's off.
Transcript
From the transcript Latent Space publishes with the episode.
- 5:17
Are they defining environments, like adding policies for reward hacking? What is the frontier of all this? If you want to hill climb in one domain, how many environments? Is it one environment with a bunch of tasks? What, what goes into it? Yeah, it's, I mean, it's all the above. So, like, one-- to produce one unit of valuable training data or of, of an eval, um, requires authoring that environment, which means there's a whole team underlying it to build the technologies to build that environment, right?
- 5:46
So for example, um, let's, let's give a concrete example. I wanna build, uh, a unit of data for, uh, an investment banking task that I'll have in an eval, and, and we'll get into way more details on that. But the gist of it, um- First, I need to have all the tools that a banker might have access to. So, um, certainly there's, like, Excel and PowerPoint, there's Outlook, there's, um, a G Drive- FactSet...
- 6:11
Bloomberg, FactSet, all these things, right? SEC Edgar. Um- Regular web searches. Oh, exa- [laughs] Uh, I can't use the real software. There's, for a bunch of different reasons, right? There's, like, rate limits, there's, like, problems necessarily, you know, potentially with the tools. So I'll build simulated versions that can spin up in my own Docker image or, or whatever when I'm, when I'm doing the task.
- 6:31
Um, which means we have people that build simulated versions of all these tools, and that's hard to do. Uh, you have to QA it, you have QC it, you have to, you know, build it from scratch. There's a whole, like, underlying ecosystem of just building these tools. Um, then you need to be able to, uh, ensure that the world that you're building has all the underlying data that would be reflective of that investment bank.
- 6:54
So sure, you have an Outlook tool, but you gotta populate it with all the email threads and, and, uh, uh, conversations that might happen. You might have a Google Drive, but you gotta populate it with the data room and with all the files that, like, the banker would need to, like, analyze, right? Um, now doing that, the optimal thing to do ideally would be you go acquire data from a bank and you de-identify everything, so, you know, there's no, you know, PII issues or anything like that. That's really hard to do.
- 7:19
It's a really, really hard problem. Um, and it's capital intensive. There's a huge time lag. And so when you think about, like, the data that labs want, they want diverse data, they want scalable data. Um, you can't just get it from, you know, buying company data and de-IDing it. So you have to, you have to generate it. If you just synthetically generate it, ask a model to build, like, a data room of files for a company, right, that could populate that G Drive, it j- it just doesn't... It's slop. It doesn't work.
- 7:47
So you need experts, like, really steering the agents and, like, iterating on it. It's like this really cool hybrid process where they're authoring that data in conjunction with an agent, um, which I think is a cool window into, like, the future of work basically across all industries, right? Authoring them so they're real-world seed. Exactly, yeah. Where we buy the data from an actual company itself and then put experts in, in the middle to help synthetically enrich that data. Yeah.
- 8:11
And it might be a, you know, smaller seed. And so, so then, um, you gotta make sure you, like, audit that environment. You have to make sure, you know, there's a bar for... There's a couple different bars, right? One bar is, hey, if I open up one of these Excel files, does it look like a financial model that a company might have had, right? Does it pass the sniff test? Um, generally, models are actually pretty good at, at generating that. But then there's a second bar, which is just, like, internal consistency across all the different files I've built and, and actual realism, right?
- 8:40
Models fail a lot on that bar. When I'm, when I'm making 20 different files, right, and I have to make sure that all these things would, would tie out the same way that it would for, like, a real company, right? There's a lot of internal inconsistencies, um, when agents just generate this from scratch. So you have to really think about, like, how are you generating this? How are you auditing it, et cetera, right?
- 8:59
And that's a big expert agent process in the loop. Yeah. Now you've built the environment. You have to make sure it can't be reward hacked or it can't be, like, you know, uh, uh, there's some- there's not something breaking. And so there's not just the expert QA process, but there's certainly just, like, you know, the, uh, um, you know, we'll run, like, a bunch of rollouts against it and try to, like, uh, uh, investigate, like, are, is the agent doing anything to reward hack the environment?
- 9:24
Um, and the expert will sort of, like, review those things before we publish an environment for tasking. Right, so that baseline is just all that has to happen before you can even start the, like, tasking part, right, in, in theory. That process is its own form of, like, authoring data, right? Now you're making the task. You have the environment. You gotta, uh, write a prompt that makes sense within the context of that environment. That's really hard to do because this is something generated.
- 9:49
Even the person authoring the data has to have, like, really good interpretability on the environment. So there's a lot of tooling, interface, et cetera, to be able to, like, understand what am I even, you know, uh, what is the environment I'm looking at, right? Um, so then you, you write a prompt that is sufficiently complex for the agent that makes sense within the environment, rollout happens, and you then author the rubric that sort of codifies what does good look like for what the agent has just done.
- 10:16
And that can be, you know, hundreds of, of rows of a rubric, right? Because there's so many different elements of what makes a good... Let's say I'm asking to build a f- you know, five-tab LBO model in that, you know, as, as the banker in that task. There's so many components of what makes a good LBO, and that can be hundreds and hundreds of rows. I have to think about the right way to weight these things.
- 10:38
I have to make sure that no individual criteria I've written in that rubric is, uh, uh, too vague such that an agent could reward hack it or too specific such that it would, you know, exclude a good answer. Um, uh, and those are just, that's an inherently iterative process. So you gotta make sure the, the, the expert in that loop has a lot of different perspectives that they can think about and, um, you know, to, to author that rubric really well.
- 11:02
End to end, these things can take, I mean, they take days. Um, and that's for one unit of data. [laughs] Right? And so now you have to do that across hundreds, uh, uh, uh, thousands of tasks. Um, usually labs don't want, you know, thousands of tasks from just one environment. They want a diversity of things, which makes sense, right, for, for how they're trying to train their model. So for any given, like, specific domain, you're talking about, like, you know, hundreds if not thousands of environments, thousands if not tens of thousands of tasks, and you gotta do that across every sub, sub, sub-domain across the economy.