The alpha, twice a week
That was 6 min.
Get the best new moments in your inbox every Wednesday and Saturday. It's free.
Check your inbox
We sent a link to your address. Click it to finish signing up. Not there in a minute? Look in spam.
or
Latent Space · 6 min · Nov 9, 2026
Labs test data by training models, not just inspecting examples
Nishant Ranka describes spot checks, model runs, ablations and embedding clusters to test a dataset.
In this episode
Full analysis- 1:25:35One goodwill error can invalidate an otherwise polished deal model5 min
- 1:46:36An agent can beat an Excel grader by listing every number2 min
- 2:11:20Loan automation can produce illegal pricing from benign inputs5 min
- 5:12Building a training task starts with the whole work environment6 min
- 15:57Labs test data by training models, not just inspecting examples6 min
- 1:31:00Expert rubrics use gates for errors that poison later calculations5 min
- 1:56:45A safe redirect can still be wrong after a child-harm request4 min
- 2:20:01As agents gain autonomy, alignment must enter the data5 min
“And if it's too incapable or too capable, then it's not in the Goldilocks zone for the model to, to learn from it.”
Takeaways
- Data can look polished but teach the wrong behavior.
- Ablations show whether a dataset helps a model improve.
- Model failures can reveal missing context in the data.
About this segment
Nishant Ranka describes spot checks, model runs, ablations and embedding clusters to test a dataset. Training results expose missing context and whether the data is diverse enough to teach a capability.
Handshake AI

Shawn Wang (swyx), Host · Co-founder, AI Engineer
Wrote “The Rise of the AI Engineer” and founded Smol AI, maker of AI News.

Alessio Fanelli, Host · Founder, Kernel Labs
Partner at Decibel, backing technical founders who build developer tools.
From “What Frontier Labs Buy: A Data Deep Dive with Handshake AI — Garrett Lord & Team”, Nov 9, 2026. Watch the full episode
Thanks. We'll look at and fix what's off.
Transcript
From the transcript Latent Space publishes with the episode.
- 16:02
Um, after you create environments, rubrics, you guys have to close the loop of evaluating them, right? What's the process of, like, are you training your own? Are you post-training? Are you across the stack? How do you test the environments? Yeah. So I've been working on labs for the last five, six years. When I started in 2021, it was like, it was much simpler because the kinds of things that the models had to do were, like, a prompt would be, uh, "Fi- help me find the best TVs on the market."
- 16:29
And then you can pretty easily cross-check that response and understand if the, the data that we are training the model on is good or not. Now it's, uh, really complex environments, so a lot of the, um, the way we test the data in labs has also become agentified, where we would write, uh, a set of, um, criteria that the data needs to meet, both on individual data sets, but as well as on the diversity of data sets.
- 16:56
And, um, what I loved to do was to, I would spend a full day just looking at one piece of data, and there was no way that I could look at every single piece of data. But just looking at one piece of data told me a lot of things that Sahil was talking about. For instance, does it have a lot of synthetic data in it? Is the instruction following templatized, where there are a lot of things where the model is being asked to generate something in ten thousand words for LinkedIn? That doesn't make sense.
- 17:25
There's nothing on LinkedIn that's ten thousand words. But we saw this in data all the time because it was generated almost, like, with a template behind the scenes, and those things are very dangerous to post-train the model on because the way, the way-- Just to, like, take a step back and talk about training, there's pre-training, where we are just taking a lot of data and showing to the model. Um, then there's mid-training and post-training, where increasingly models become aligned to the tasks that they will do in the real world.
- 17:52
And if you show the models the tasks that are misaligned, um, for instance, that have these templates in it or synthetically generated but are not high quality, then the models, uh... You do the whole training run. Sometimes the, the loss curves look great. The, the rewards in the training curve look great. You take the final checkpoint, and you start playing around with it, and it makes no sense.
- 18:17
It's either, uh, thinking for way too long, or it just, like, keeps on writing as, uh, someone who in real profession would never write. So as a result, when we look at the data, we are very concerned about its ability to actually train the model. And how do we close that gap between the data is high quality and understandable, but can it train the model? And that's what, like, a lot of what labs look for when they're looking at the data.
- 18:42
So other things we would do is, uh, just, like, test it out on a model and see how many times is the model capable of actually passing the data. And if it's too incapable or too capable, then it's not in the Goldilocks zone for the model to, to learn from it. Similarly, we would, um, do ablations where we'll take a set of data, train the model a little bit on it, and see how things change. Maybe take an in-between checkpoint and start dogfooding it.
- 19:09
Another thing we, we would love to do is, um, take the whole data, um, embed it, and then pick up different clusters to see actually it's diverse. Is it all talking about if it's finance data, or just talking about one type of M&A? Or is it representing wealth management, it's representing corporate tax, it's representing all the different things that we expected in the data? So it's a, it's a very messy process. It's, uh, uh...
- 19:35
You're, like, trying to somehow figure out, is this data good? And it's all under very high time pressure because the time between collecting the data, post-training the model, and launching model is really compressed, maybe a few months at the best. So we want partners who have shown consistent promise of delivering high-quality data because there's not very good ways to test the high-quality data under the time pressure.
- 20:01
Um, and you were also asking a little bit more about... You should talk about some of the post-training efforts we do internally. Yeah. I mean, the... one of the things that I'm doing and, and Handshake has been doing is taking all this data, post-training it internally to do two things. One- What, what do you post-train it on? Like open models or like- We take open source models and post-train it on our own data, and the big thing we are seeing on the benchmark we have built and the data that we have put together, can the data help train some of these small open source model to climb this benchmark?
- 20:30
And there are two interesting things from that. One is that how is it climbing or not climbing? So for instance, if we see that the model has a visual task and a reasoning task associated with it and, and the model doesn't climb, is it because it failed to understand the image? Or is it that it failed to understand the, um, reasoning on how to reason the, the description of the image? And based on what we learn on this post-training, we can go back to the data team that Sahil leads and say like, "Hey, we think that all these images need to come with a description," or, "All these financial models need to come with notes from the investment banker saying what this re-represents," or maybe, "This tax law needs to come with, like, attached PDFs that just describe this tax law," because the models cannot find this or they get confused on the web.
- 21:19
So we can isolate the, the requirement. So that's one big part of post-training, is to, like, get all these insights on data. And then the other part is that if we put all this together, can we actually help the model climb? Because if the model does climb, then it's very strong proof point to the labs as well as the enterprises who are trying to fine-tune their own models that this data is actually good, because the purpose of data is to train models.
- 21:44
So we, we try to complete the whole loop. Like, two last quick questions before we go into environments. One is, how much data do you need? So say there's a company that, you know, they're well-established.