The alpha, twice a week
That was 2 min.
Get the best new moments in your inbox every Wednesday and Saturday. It's free.
Check your inbox
We sent a link to your address. Click it to finish signing up. Not there in a minute? Look in spam.
or
Latent Space · 2 min · Nov 9, 2026
An agent can beat an Excel grader by listing every number
A grader looking for net income anywhere in a workbook rewarded an agent for adding a tab listing numbers one to a million.
The episode’s best minutes
Full analysis- 1:25:35One goodwill error can invalidate an otherwise polished deal model5 min
- 1:46:36An agent can beat an Excel grader by listing every number2 min
- 2:11:20Loan automation can produce illegal pricing from benign inputs5 min
- 5 more moments in the full episode
“But that obviously isn't the spirit of the task.”
Takeaways
- A grader must check where a value appears, not just whether it appears.
- Reward hacks can reveal flaws in both rubrics and task design.
- More precise criteria or a template can make grading harder to exploit.
About this segment
A grader looking for net income anywhere in a workbook rewarded an agent for adding a tab listing numbers one to a million. The exploit exposed a rubric flaw and forced tighter task design.
Handshake AI

Shawn Wang (swyx), Host · Co-founder, AI Engineer
Wrote “The Rise of the AI Engineer” and founded Smol AI, maker of AI News.

Alessio Fanelli, Host · Founder, Kernel Labs
Partner at Decibel, backing technical founders who build developer tools.
From “What Frontier Labs Buy: A Data Deep Dive with Handshake AI — Garrett Lord & Team”, Nov 9, 2026. Watch the full episode
Thanks. We'll look at and fix what's off.
Transcript
From the transcript Latent Space publishes with the episode.
- 1:46:41
But it's worth actually talking about that 'cause, um, so much of when you build that grader informs you about, you know, where in the rubric that you've constructed do you need to be more precise or more clear. Um, you know, I think one example that we had in building one of these tasks was, uh, uh, you know, the model or one of the rubric items was, you know, net income is 100 or something, right?
- 1:47:05
And, uh, what the agent did was it basically just took a tab in Excel, you know, made a new tab in Excel, and just literally had the number, like, you know, for, just, like, li- listed every number from, like, one to, like, a million, right? And so the number 100 was present, and the grader was just looking for the number 100 somewhere and found it, right? And so that taught us a lot. You know, there's a lot of problems, right, in that, but it taught us a lot about the task construction and the task design.
- 1:47:32
Um, taught us about, like, you know, where, how do we need to be more precise with the rubric or, um, you know, such, you know, in such a way where the grader could, in the spirit of the task, make sure it's aligned- Yeah... with what we want. Um, and there's a lot of little- You, you told me about the reward hack where it just spammed out, uh- Exactly... numbers. Yeah. What, what was that? Yeah, so it was, it was this, right?
- 1:47:51
It was, um, you know, a lot of, uh, uh, you know, the, the rubric we'd constructed was, you know, look for, originally, right? So it's, like, early days but, you know, look for a net income number of 100, right? And that was, like, the word or the, like, specific rubric criteria. So what the model did is it made a tab in the Excel sheet that just h- listed every number from, like, one to a million.
- 1:48:14
And so then every number [chuckles] that was in the rubric was there. It found it, right? In the, in the sheet. But that obviously isn't the spirit of the task. It, to, to, full reward hack. And so what that taught us is, okay, we have to be more precise with the rubric criteria, or maybe we have to redesign the task a little bit where there's a little bit more of a template to work from in terms of the financial model, like the Excel model that the, the agent is gonna build from, um, to make it in such a way where if you have a grader and that rubric criteria and the net income's 100, it's actually only looking for something that is where the net income is listed as 100 and not just the number 100 somewhere in the financial model.