# Verifiability helps explain where AI excels

Markdown copy of https://realalpha.app/s/verifiability-helps-explain-where-ai-excels/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Andrej Karpathy](https://realalpha.app/builders/andrej-karpathy/) · Andrej Karpathy is co-founder of OpenAI, former head of AI at Tesla, and founder of Eureka Labs. · Joined Anthropic's pretraining team; founded Eureka Labs, an AI-native school, after leading Tesla's Autopilot AI.
- Host: Stephanie Zhan · Partner, Sequoia
- Show: Training Data
- Episode: [Andrej Karpathy: From Vibe Coding to Agentic Engineering w/ Stephanie Zhan](https://realalpha.app/v/96jN2OCOfLs/), Apr 29, 2026
- Moment: 9:29 to 13:40 of the episode (4 min), playing on https://realalpha.app/s/verifiability-helps-explain-where-ai-excels/; the episode from there: https://realalpha.app/v/96jN2OCOfLs/?t=569

## Takeaways

1. Models improve fastest in areas where their output can be verified.
2. Training data can create sharp strengths beyond general capability gains.
3. Test whether your task falls within a model's strong circuits.

## About this segment

Andrej Karpathy links models' jagged abilities to what labs train and reward, contrasting code and chess with a simple car-wash question. That unevenness is a reason to test models in the specific tasks you need.

From “Andrej Karpathy: From Vibe Coding to Agentic Engineering w/ Stephanie Zhan”, Apr 29, 2026. [Watch the full episode](https://realalpha.app/v/96jN2OCOfLs/)

## Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[9:32](https://realalpha.app/v/96jN2OCOfLs/?t=572) but I think we're going to probably get there sort of piece by piece. And that progression is TBD, I would say. I'd love to talk a little bit about this concept of verifiability, the fact that AI will automate faster and more easily domains where the output can be verified. If that framework is right, what work is about to move much faster than people realize, And what professions do we have that people actually think are safe?

[10:00](https://realalpha.app/v/96jN2OCOfLs/?t=600) that are actually highly verifiable. Yeah, so I spent some time writing about verifiability and basically like traditional computers can easily automate what you can specify in code and kind of this latest round of LLMs can easily automate what you can verify in a certain sense. Because the way this works is that when Frontier Labs are training these LLMs, these are giant reinforcement learning environments, so they are given verification rewards, and then because of the way that these models are trained, and they end up basically progressing and creating these jagged entities that really peak in capability in verifiable domains like math and code and adjacent, and stagnate and are a little bit rough around the edges when things are not in that space.

[10:45](https://realalpha.app/v/96jN2OCOfLs/?t=645) So I think the reason I wrote about verifiability is I'm trying to understand why these things are so jagged. And some of it has to do with how the labs train the models, but I think some of it also has to do with the focus of the labs and what they happen to put into the data distribution. Because some things basically are significantly more valuable in economy and end up creating more environments because the labs wanted to work in those settings.

[11:06](https://realalpha.app/v/96jN2OCOfLs/?t=666) So I think code is a good example of that. There's probably lots of verifiable environments they could think about that happen not to make it into the mix because they're just not that useful to have the capability around. But I think, to me, the big, I guess, like the big mystery is the favorite example for a while was that how many letters R are in a strawberry and the models would famously get this wrong.

[11:25](https://realalpha.app/v/96jN2OCOfLs/?t=685) and it's an example of jacketness. The models now patch this, I think, but the new one is I wanna go to a car wash to wash my car and it's 50 meters away, should I drive or should I walk? And state-of-the-art models today will tell you to walk because it's so close. How is it possible that state-of-the-art Opus 4.7 will simultaneously refactor 100,000 line code base or find zero-day vulnerabilities and yet tells me to walk to this car wash?

[11:53](https://realalpha.app/v/96jN2OCOfLs/?t=713) This is insane. And to whatever extent these models remain jagged, it's an indication that, number one, maybe something slightly off, or number two, you need to actually be in the loop a little bit, and you need to treat them as tools, and you do have to kind of stay in touch with what they're doing. And so I think all of my writing, long story short, about verifiability is just trying to understand why these things are jagged. Is there any pattern to it? And I think it's some kind of a combination of verifiable plus labs care. Maybe one more anecdote that is instructive is from GPT 3.5 to GPT 4.

[12:30](https://realalpha.app/v/96jN2OCOfLs/?t=750) People noticed that chess improved a lot, and I think a lot of people thought, oh, well, it's just a progression of the capabilities, but actually it's more that, I think this is public information, I think I saw it on the internet, a huge amount of data of chess made it into the pre-chaining set, and just because it's in the data distribution, basically the model improved a lot more than it would just by default.

[12:51](https://realalpha.app/v/96jN2OCOfLs/?t=771) So someone at OpenAI decided to add this data, and now you have a capability that just peaked a lot more. And so that's why I think I'm stressing this dimension of it, as we are slightly at the mercy of whatever the labs are doing, whatever they happen to put into the mix, and you have to actually explore this thing that they give you that has no manual. And it works in certain settings, but maybe not in some settings, and you have to kind of explore it a little bit. And if you're in the circuits that were part of the RL, you fly, and if you're in the circuits that are out of the data distribution you're going to struggle and you have to kind of figure out which circuits you're in in your application. And if you're not in the circuits then you have to really look at fine-tuning and doing some of your own work

[13:32](https://realalpha.app/v/96jN2OCOfLs/?t=812) because it's not going to necessarily come out of the LLM out of the box. I'd love to come back to the concept of jagged intelligence in a little bit.