The alpha, twice a week
That was 6 min.
Get the best new moments in your inbox every Wednesday and Saturday. It's free.
Check your inbox
We sent a link to your address. Click it to finish signing up. Not there in a minute? Look in spam.
or
Lightcone · 6 min · Nov 14, 2026
Agent review can overfit without product context
Lauren Tan says multi-model review can flag issues without understanding product intent, so it may overfit.
In this episode
Full analysis- 16:09Bots should earn broader access as people learn to trust them7 min
- 40:06Agents improve when they can run and check their own code3 min
- 52:17Product direction becomes the bottleneck as execution gets cheap4 min
- 43:54Agent review can overfit without product context6 min
- 22:26Slack can carry a feature from request to shipped code3 min
- 35:18Cheap execution shifts the bottleneck to defining goals5 min
- 6:26Bot roles work best when each owns a focused problem4 min
- 3:35Marketing teams can connect bots to operational workflows3 min
“I think you run the risk of overfitting where the sort of agent doesn't have the ability or maybe enough judgment to determine what's actually good feedback”
Lauren Tanat 44:57 · LightconeTakeaways
- Review agents may not know which feedback fits the product’s goals.
- Static analysis and lint rules can catch recurring issues cheaply.
- Keep humans involved for critical infrastructure and deployment changes.
About this segment
Lauren Tan says multi-model review can flag issues without understanding product intent, so it may overfit. She favors encoding repeatable constraints in architecture, static analysis and CI.
Principal engineer, SpaceXAI
Created pstack and co-authored React Compiler 1.0; now builds Grok Bot's agent workflows and runs 20+ agents.

Garry Tan, Host · President & CEO, Y Combinator
Co-founded Posterous and Initialized Capital, an early Coinbase backer.

Harj Taggar, Host · Managing partner, Y Combinator
Co-founded Auctomatic with the Collison brothers and later Triplebyte.

Diana Hu, Host · Managing partner, Y Combinator
CTO of Escher Reality; after Niantic bought it, ran AR for Pokémon GO.

Jared Friedman, Host · Managing partner, Y Combinator
Co-founded Scribd, which YC funded in 2006; has advised 20+ YC unicorns.
From “The Rise Of AI Coworkers”, Nov 14, 2026. Watch the full episode
Answers “Are AI agents ready for real work?” See all 157 answers
Thanks. We'll look at and fix what's off.
Transcript
Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.
- 44:00
of like a review step then. How do you, what's your philosophy on like, how much review is enough versus too much? Like, I'm literally in a debate with one of my cappy captains right now, where it's like, I wanted to go from 50 million tokens for review to like 10 million. But it, it has all these held out evals that show that, you know, we have like 30 types of bugs that I'm trying to make sure I can find in that review step. And it's like, I can't do it with 10. I need 30. Like, I went from 50 to 30. Can we shipped 30 and i'm like no no you need 10 what do you see like i mean what's the state of the art really good question i think so for me actually in pstack i have tried to uh downplay some of the review skills a little bit you know in pstack we have a bunch of skills like there's one called
- 44:45
interrogate and that will spawn uh agents of different models so like you know like across different families which works really well you can do like adversarial review is that was like quite hot many months ago. I've actually kind of withdrawn a little bit from that because I find that when you do that, I mean, it's still valuable, but I think you run the risk of overfitting where the sort of agent doesn't have the ability or maybe enough judgment to determine what's actually good feedback versus not good feedback. And it maybe doesn't have complete knowledge of your goals and your intents and your wider product vision. It doesn't understand how you want the product to evolve for example and so you can get to a point where you are you know very abstractly you know it's kind of like imagine before agent you have a bunch of engineers talking
- 45:33
about the the most perfect ideal utopia way of doing this but actually you know that doesn't even matter because in in two months we plan to deprecate this thing and build a completely new vision right and so i think there's some analog similar analogy there where i think some review is good but i try to move more towards um like on my screen here i have this inverted pyramid where you know like i try to push like you mentioned you know spending less tokens and so from the engineering point of view uh i like to think of this as you know your your code base is a form of memory and so it's much more token efficient if you know you refactor your architecture your api design uh or you have you know things like static analysis uh you know lint rules compilers all of these you know languages programming languages these all exist to help
- 46:28
you know in the past like humans write good code and unsurprisingly they also help agents right and so when you can encode some of these things that you are you know you know arguing with your captain about back into the code base. That's just like one less problem you now have to think about. You just gave me a feature idea, which is like, because G-Stack has office hours and like the CEO review where you're like talking about the product, I can use that.
- 46:57
I mean, because those are like product requirements that feed in and those save to markdown in the project. So I could have the review skill. You could combine them, yeah. I think it already does this a little bit, but I could be like, no, no, no, really use that to figure out what we should review and what counts as a real bug to fix versus not. Because there are, like, general correctness bugs that are just, like, they don't matter.
- 47:18
Like, they might only happen, like, 0.1% of the time or maybe never. So it sounds like this triangle shows us that, you know, the code base is quite large and then, like, you have very little time. Yeah, I think that's why I intentionally drew it this way. uh i guess it's not a very scientific chart of course but i the way i imagined it when i drew it it was it was i think like this is in terms of where you should spend your time as an engineer so for me you know like most of the investment i make is into the code base and in static analysis and i still layer that with you know rules and skills you know pstack updates and agents.md and we still do kind of like a little bit of human review from from time to time i think you know if you're working on like more critical parts of like infrastructure or databases deployment you know like kind of more uh you know sensitive things
- 48:11
then you might actually still want to stay in the loop um but i think a large class of your regular product development or software maintenance like an investment here actually ends up saving you a lot of tokens uh because like for example you know with the the grokbot desktop app uh it's an electron application and one of the lessons we learned from building cursor is that the renderer thread on electron is very sensitive uh to long tasks yeah yeah so like you can get very janky performance if your tasks take more than like 16 milliseconds or eight milliseconds right if you want to hit a 60 fps smoothness we literally just built that when we were refactoring and building robot we decided we don't want to think about this problem anymore we don't want to have to catch it in code review we'll actually build it as like a lint rule we built a dependency graph
- 49:03
checker uh and now it's just enforced by the code base enforced by ci so it's tokens i don't have to spend anymore it's somehow enforced in static analysis actually like can know how long something will take to run oh no it's more like so uh for the graphite app we have code that runs in the main thread and then the renderer thread. And so we explicitly forbid importing between those two things. So it's not necessarily that, you know, main code thread will always be long.
- 49:34
It's just, like, oftentimes, you know, some, like, heavier tasks that shouldn't, you know, block the renderer. Never do long work on the renderer thread, man. Yeah, you want your renderer to be as light as possible, you know? So, yeah, that's something we just don't think about anymore. And I think the performance of the app has actually stayed very consistent and very good, which is I think is actually quite surprising for a lot of like a majority AI written applications where, you know, they start out fast and then over time they just accumulate a lot of slop and performance problems.
- 50:10
But with the right skills and rules. With the right setup, I think you can actually, you can have quality and you can have quantity. I think you can have your cake and eat it too.