Try Pro free

Latent Space · 5 min · Nov 9, 2026

Expert rubrics use gates for errors that poison later calculations

Collin Schweiker explains how expert rubrics combine numeric checks with construction criteria and shut off credit when a critical step fails.

In this episode

Full analysis
  1. 1:25:35One goodwill error can invalidate an otherwise polished deal model5 min
  2. 1:46:36An agent can beat an Excel grader by listing every number2 min
  3. 2:11:20Loan automation can produce illegal pricing from benign inputs5 min
  4. 5:12Building a training task starts with the whole work environment6 min
  5. 15:57Labs test data by training models, not just inspecting examples6 min
  6. 1:31:00Expert rubrics use gates for errors that poison later calculations5 min
  7. 1:56:45A safe redirect can still be wrong after a child-harm request4 min
  8. 2:20:01As agents gain autonomy, alignment must enter the data5 min
“we intentionally create the environment, the inputs, um, to be, you know, there's only one right answer, right?”
CSCollin Schweikerat 1:32:08 · Latent Space
1:32:08

Takeaways

  1. About half the rubric checks numeric outputs.
  2. Tasks are designed to make defensible methods explicit.
  3. A critical error can zero out dependent criteria.

About this segment

Collin Schweiker explains how expert rubrics combine numeric checks with construction criteria and shut off credit when a critical step fails. The gates reflect how one error can contaminate downstream calculations.

CS

Collin Schweiker

Handshake AI

  • Shawn Wang (swyx), Host · Co-founder, AI Engineer

    Wrote “The Rise of the AI Engineer” and founded Smol AI, maker of AI News.

  • Alessio Fanelli, Host · Founder, Kernel Labs

    Partner at Decibel, backing technical founders who build developer tools.

From “What Frontier Labs Buy: A Data Deep Dive with Handshake AI — Garrett Lord & Team”, Nov 9, 2026. Watch the full episode

Transcript

From the transcript Latent Space publishes with the episode.

  1. 1:31:04

    Two is there's a lot of work here that's done for thirty minutes that is correct. In this case, there's one error. Sometimes there's multiple. How do you transition that out into useful reward signal? Because you did a lot of inference and a lot of it's correct. Um, maybe start off with the rubric of how this is graded and Answer as you- Yeah. So creating the rubric is by far the most challenging part of, you know, these semi-verifiable domains.

  2. 1:31:30

    Generally, it's very easy or, or, or straightforward to say, like, "This output is a 0%," you know, they return absolutely nothing, like a blank page or fa- okay, under- understood. Or, "This output is 100%," like, "Oh, they, they nailed it perfectly, and I, you know, I would, I wouldn't change anything. This is perfect." What's really hard is every step in between, particularly when there's, there's multiple variables at play, right? Maybe, maybe Model A is better at, you know, uh, the, the cashflow build down, but Model B is better at the balance sheet. Well, how do you, how do you, uh, how do you balance those two things, right?

  3. 1:32:04

    And so we spend a lot of time building these rubrics with experts, and the goal is to make them as deterministic as possible, right? So we, we intentionally create the environment, the inputs, um, to be, you know, there's only one right answer, right? And, you know, there may be conventions in the economy or the, you know, the industry where there, there could be two defensible or reasonable, uh, you know, approaches, but we just-- we a- you have to make absolutely sure in the environment that, okay, there are two reasonable approaches, but actually this note from Sahil says, you know, "Use this approach," right?

  4. 1:32:42

    So then we can, we can grade it on the rubric. So that's how we, that's how we approach the rubrics. Generally, the rubrics are about 50%, uh, just hard checks, so this is like a simplification. Mm-hmm. But, you know, ha-half of the rubric is, is the numbers. Like is, you know, is, is the year one EBITDA right? Is the year two cash balance right? And we think that's reflective and that is reflective of the industry standards. Like, you could argue it's even higher. So at a high level, it's like the key steps of solving a task.

  5. 1:33:13

    There's some binary answer you should have step one to five, you should have this. That's just binary yes or no. Yeah. So yeah. So we, we grade the actual n-numerical outputs- Yeah... for, you know, ha-half of the, the points or criteria. And, and in general, in, in our benchmark, there's, I think, on, on average, 146 different criteria, and then the remainder of the, the points are allocated across construction. So not necessarily, um, you know, the numbers, but, but really drilling down into, uh, the process.

  6. 1:33:46

    And so this, this is the gate that, that we've been talking about, and this is, you know, I would also consider a construction output because, you know, the step-up is built as purchase price plus estimated working capital liabilities, less tax bases and current assets, less tax bases and appreciable assets, and recovered straight line over 15 years. This is, this was discussed and shown in the environment multiple ways. You know, we knew that the agent looked at the source material and this is, you know, fair to grade it on, and it got it wrong.

  7. 1:34:16

    Again, because, you know, something in, you know, its, its weights to, to, to looked at what it was calculating and forgot, you know, that this is a special exception and, and kind of maybe reverted to its treatment of goodwill for other scenarios. Um, but that's how we think about the rubric, and we can use the rubric... And so, sorry, this gate shuts off everything else in this section. It's just like a, a VP would, right?

  8. 1:34:41

    If I, you know, if I'm showing, if I'm going through a model, uh, you know, with my managing banker or whatever, and, you know, row nine is wrong, like they ch- you know, they check everything else, but I-- once they get to row nine, that's wrong, you know, there's a bust in your model or there's a gremlin or something, you know, y- they circle it and send it back, right? They don't just ke- they don't keep checking because, number one, they've lost trust in you because they, you know, they see this error and it's clearly you, you didn't get this right, so why are you gonna get everything else right?

  9. 1:35:11

    Mm-hmm. Mm-hmm. But also because these errors compound. Again, this is, you know, this is over 2,000 interconnected formulas, and so if there's one error, that error pollutes every other number in it. So and because it's realistic, we shut off the gate for every, you know, all 180 points in the section dealing with the tax mechanics. And so we can use this rubric, which again, has a lot of specificity on, on the numbers, and you can, you can actually just logically go through.

  10. 1:35:38

    Well, it actually got, you know, years one, two, and three correct for these numbers. It got years four wrong, so now you can, you can direct your attention to what happened between year three and year, year four that there, there's an error there, right? And so we can use, we can use the rubric and we use the trajectory to come up with kind of the, the failure modes and understand which models are strong and weak in, in various types of tasks.

  11. 1:36:02

    Um, the three biggest issues that we saw across models for this benchmark were financial logic, where it was unable to, um, unable to, you know,