Free with a sign-in.
The best 15 min of 61
Jev Creator on Decision Models, Benchmark Fatigue & Why the Labs Missed It
Diogo Almeida argues that AI needs reliable, machine-native decisions before it can automate more routine work.
Unsupervised Learning · Jacob Effron
- Diogo Almeida is creator of Jev and cofounder of TypeSafe AI.
The best minutes
- 17:57Reliability matters more than inventing new model shapes4 min
- 26:55The KV cache constrains how coding agents are designed8 min
- 52:49Coding agents may speed engineering by 20 to 40 percent3 min
Also worth watching
- 1:16Most model calls should serve software, not people5 min
- 21:55Semantic checks could make agent software safer5 min
- 34:39Multimodal models are less reliable than text models3 min
- 48:52The labs may be strongest at financial engineering4 min
- 55:49True AI agency may require two major breakthroughs4 min
Skip the discussion of model categories, the launch and benchmark recap, and the fundraising exchange.
Full analysis · 6 min read
Real Alpha · Nov 7
Diogo Almeida says coding agents speed engineering 20 to 40%
- Almeida estimates coding agents speed engineering by 20 to 40%, depending on task and user.
- He says reliability takes more work than new model shapes, while benchmark hype can erode developer trust.
- Teams should test the speedup in their own workflows, counting review, rework and side effects.
Free with an account
Read the rest of the analysis
7 sections, the 6 other voices it tests Almeida against, and what it means for you. Every episode we cut gets one.
Check your inbox
We sent a link to your address. Click it to finish signing up. Not there in a minute? Look in spam.
or
Signed up before? The same buttons sign you in. The episode, its moments and the transcript stay free.
Software calls should serve machines, not readers
Almeida's central mismatch is not a shortage of intelligence, but a gap between capability and productivity. He asks what share of model calls should serve people rather than software. He says even text-focused listeners put an overwhelming share of imagined calls on the machine-native side.
That is his design premise, not a measurement of current traffic. The host described today's calls and tokens as more human-shaped. Almeida answered that the relationship is an interactive dance, with models and use cases co-evolving.
A response designed for human reading can prioritize style and prose. Software often needs a structured decision it can consume. For example, Almeida likes chatbots for pseudo-expert answers that need not be correct, and coding agents for disposable dashboards.
Those uses tolerate rough edges more readily than accounting automation. His contrast is pointed. He asks why models can solve Millennium Prize problems but cannot automate work touching accounting.
Our read
The unit of progress is not a striking answer but a decision a program can reliably use. That makes interface and workflow design part of model usefulness, not decoration.
Reliability matters more than new model shapes
Almeida says new task shapes can be relatively quick to make, while reliability is the harder work. Benchmark wins and launch demos can display capability without showing whether software succeeds repeatedly.
His concern is adoption as well as engineering. When releases overpromise and disappoint, developers become less willing to evaluate the next one. That makes hype costly even when a genuinely useful model appears later.
Almeida created Jev and cofounded TypeSafe AI, so his reliability focus is also tied to the product he is building. The interview gives no independent, side-by-side results proving Jev is more reliable than competitors. His criticism is a testable product thesis, not comparative proof.
Dylan Field made a related distinction on All-In in October 2026, saying model progress can be mistaken for product progress. That supports the distinction, but does not validate Almeida's reliability claims.
“the shape really doesn't matter compared to actually it being useful”
Our read
Almeida's warning about benchmark fatigue is credible as a product risk, not proof of Jev's relative performance. Buyers still need task-level evidence before trusting any provider's reliability story.
Developers know the reliability their workflows require
Almeida says application developers know better than providers what reliability profile a downstream task needs. Providers can expose useful controls, but they do not know the local cost of every error.
That makes a single model ranking an incomplete buying guide. He sketches semantic dependent types that could verify properties beyond a basic type, such as an integer greater than zero. He also proposes fast semantic checks for each tool call, tailored to an organization's rules.
These are proposals in the conversation, not demonstrated product features. Our practical translation is to document failure cases and ask providers to address specific weaknesses. Then teams can retest those failures in the real application.
Our read
A generic benchmark cannot specify what an application must never do. Our recommendation is to bring concrete failure cases to providers and measure any fix in the actual workflow.
The KV cache limits some agent designs
Almeida says coding-agent interfaces are shaped by KV-cache economics, especially append-only state. He argues that routing and sub-agents become harder when context must persist across calls. Passing everything forward raises input costs, while passing too little risks losing relevant information.
Repeatedly appending information can also clutter context and make models perform worse over time. The scope matters. Almeida says routing can work for task-in, task-out calls, where state can be discarded. His objection is to treating that pattern as a universal fit for stateful work.
Harrison Chase said on 20VC in October 2026 that vertical-agent routing needs task and harness context. That complements Almeida's focus on state, but does not settle his KV-cache constraint. Angela Jiang said on Latent Space in October 2026 that a lead agent can delegate parallel, asynchronous work.
Her pattern describes orchestration, not proof that agents can freely share one live cache. For example, Almeida imagines a read-only observer producing a live visualization while another agent works. He also suggests task-aware compaction and reusable project history, both architectural ideas rather than validated solutions.
Harrison Chase · 20VC · Oct 17Useful model routing needs to understand the agent harness
Angela Jiang · Latent Space · Oct 27Parallel agents need shared context and asynchronous work
Our read
The KV cache is a useful design constraint to investigate, not a settled limit on agent architecture. New state patterns still need to show that they improve work without making it less predictable or more expensive.
Almeida says image capability trails text sharply
Almeida says longer context and added modalities tend to degrade reliability. He estimates his image model is maybe 10% of the way to his text model. He calls it insufficient and says he would be embarrassed to call it production-ready.
He expects text tasks to be automated before image tasks. Emmanuel Ameisen described Claude moving from 80% to 99% accuracy on his coding in Lenny's Podcast in October 2026. That is one user's experience, not evidence for Almeida's image estimate or a broad cross-modal rule.
Still, it illustrates why small reliability changes can alter whether a workflow becomes useful. A strong result in text does not establish dependable image or computer-use behavior. Teams testing multimodal work need evidence from their own inputs and failure modes.
Our read
Progress across modalities should not be treated as uniform. Measure the workflow being automated, especially when mistakes are costly or hard to reverse.
Almeida estimates a 20 to 40% speedup
Almeida says he has error bars because long-term effects may include skill degradation and technical debt. His estimate is a 20 to 40% speedup range, but it depends on the task and the user's skill. He adds that isolating side effects in a legible way can affect the result.
Some uses let people do work they otherwise could not. Others can be as bad as coding by hand. His average judgment is more restrained. He guesses at a decent but small boost and says he believes that matches the consensus he has seen.
Guillermo Rauch said on Sourcery in October 2026 that engineers should manage agents, not blindly ship their code. Danila Shtan said on Unsupervised Learning in November 2026 that adoption depends on reliably reaching outcomes. Neither statement independently verifies Almeida's percentage.
Together, they point to an important denominator: useful output after review, correction and risk. Side effects are not the only determinant of speed, but Almeida says isolating them can help capture the gains.
“my estimates are in the 20 to 40% speed up range”
Guillermo Rauch · Sourcery · Oct 16Engineers should manage agents, not blindly ship their code- DSDanila Shtan · Unsupervised Learning · Nov 2Agent reliability may matter more than model size
Our read
Treat the range as a local hypothesis, not a company-wide productivity promise. Count accepted work alongside review time, rework, technical debt and side effects.
Almeida guesses true agency remains breakthroughs away
Almeida distinguishes human misuse of powerful tools from autonomous agency. He guesses true agency is something like two breakthroughs of gigantic magnitude away, though one gigantic breakthrough might suffice. That is his forecast, not a safety guarantee, and he acknowledges uncertainty.
His distinction is useful because misuse, bio risk, cyber risk and autonomous behavior have different threat models. Controls for a person using a tool cannot simply be replaced by predictions about machine agency. Nor does a distant agency forecast make current tool permissions or reversibility less important.
Our read
Almeida's forecast is uncertain, but separating these risks helps avoid treating AI safety as one question. Build controls around current capabilities and actual consequences.
What it means for you
- Choose one workflow. Our suggestion is to pilot a bounded task and record the baseline before adding an agent. Compare completed work with the current process, not a model benchmark.
- Count review costs. Our recommendation is to track accepted code, review time, rework and unwanted changes. This tests Almeida's 20 to 40% range against your own tasks and skill mix.
- Test context routing. Our suggestion is to compare task-in, task-out calls with stateful workflows, where Almeida says the KV cache complicates routing. Record when context transfer improves outcomes or creates failures.
- Check tool boundaries. Almeida sketches semantic checks for tool calls as an idea, not a shipped guarantee. Our suggestion is to identify sensitive actions and test whether checks block harmful effects without stopping useful work.
What to watch
- Whether independent evaluations move from benchmark scores to repeated task-level reliability.
- How much of the estimated engineering gain remains after review, rework and technical debt.
- Whether new agent state designs support shared work without degrading context.
Send this to engineering leaders setting coding-agent pilot metrics.
Watch the minute where Almeida gives his 20 to 40% speedup estimate.
Written by Real Alpha from the episode's transcript and the other moments it links, then checked against the transcript. Quotes are the speaker's exact words.
Minute by minute
- 1:06AI needs machine-native software
Diogo Almeida says he was motivated by the contrast between AI’s apparent intelligence and its limited effect on productivity. He argued that software should work backward from machine-native model calls: computers usually need bits or structured decisions, not prose. He says Jev is not yet at that goal.
- 6:34Reliability drives Jev’s roadmap
Almeida says Jev’s developers need to judge what reliability their applications require, so the company exposes model weaknesses and gives developers more control. He says real-world feedback is still limited and developers should push providers harder to address model “jaggedness.”
- 10:54Models should be provider agnostic
Almeida calls it a bad outcome if developers must build systems around fixed assumptions about particular models. He wants model behavior to become reliable enough to work across providers, like SQL. He says Jev should be a small, composable software primitive, while LLMs remain useful for answers that need not be correct and for throwaway software.
- 15:53Reliability matters more than shape
Almeida hopes LLMs—or something better—will remain useful, but says people overuse them and should keep a range of models in their toolbox. He expects more model shapes, but says creating a shape is easier than making it useful and reliable. He criticizes benchmark-driven launches for encouraging overpromising and making it harder to identify genuinely useful models.
- 21:13Software and models must co-evolve
Almeida says progress requires both better models and education that helps developers build more capable software. He points to coding agents less constrained by the KV cache, semantic dependent types that verify properties expressed in natural language, and fast semantic checks on tool calls as promising areas.
- 26:55The KV cache constrains agents
Almeida says coding-agent design often follows the KV cache’s append-only state, which limits routing and makes subagents’ context costly and difficult to manage. He describes alternatives such as agents sharing context, live visualizations of another agent’s work, and task-aware memory and compaction. He says the cache is useful; the problem is treating it as the only design.
- 34:39Multimodal models need more work
Almeida says multimodality tends to reduce reliability and expects text tasks to be automated before image tasks. He describes Jev’s image model as insufficient for production and estimates it is about 10% as far along as the text model. He is especially interested in video games and reliable computer use.
- 39:24Older models could automate easy work
Almeida says building Jev five years earlier would have been harder but possible. He argues that GPT-3.5-era models, and possibly GPT-3, were already smart enough to automate many easy tasks. He cites GPQA performance around 2023 as an example of strong measured ability that had not translated into enough real-world automation.
- 41:55Almeida says benchmark releases burn trust
Almeida says similar releases from other labs risk damaging trust when they are optimized for benchmarks rather than real-world use. He argues that lower prices alone do not expand demand: a product must also create enough value for people to use it more.
- 44:04Almeida prioritized helpful investors
Almeida says Jev can run profitably and that the company did not need its recent funding to operate. He says the goal was to bring in valuable people who could help build the business, rather than subsidize the service to create lock-in.
- 45:23Reliability beat benchmark hype
Almeida says the team considered releasing a benchmark-focused version but chose to prioritize reliability. He is glad they avoided another overpromising launch, though he says he cannot know whether it would have attracted more attention. He says developers’ fatigue with benchmark-heavy releases came sooner than he expected.
- 48:11Almeida sees financial engineering as labs’ strength
Almeida says coding agents can be valuable but questions whether their current use justifies the money being spent. He says he is not informed enough to judge Anthropic’s valuation, and sees financial engineering—not models, product, or distribution—as the labs’ greatest strength. He also worries that some expectations assume AI will deliver far more than it has demonstrated.
- 51:52Deployment depends on managing risk
Almeida says coding and math have safeguards, such as human review and version control, that can limit unwanted effects; many other kinds of knowledge work do not. He estimates coding agents deliver a 20 to 40% speedup, depending on the task, the user’s skill, and how well side effects are contained. He says some uses enable work that otherwise would not happen, while others are as slow as doing it by hand.
- 55:49Agency remains far off
Almeida separates risks from ASI, bio, cyber, and trust and safety, and says reliable systems with true agency are still about two major breakthroughs away. He distinguishes AI agency from people using powerful tools to cause harm. At the close, he urges builders to regain a sense of agency and rethink software and interfaces.
Published Nov 5, 2026.
Transcript · 54 min read
Unsupervised Learning, hosted by Jacob Effron, with Diogo Almeida.
Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.
As you think about Anthropic, is that something you're optimistic on? It's like, hey, that's undervalued or something you're like, oh, I don't know what's going to happen to that. I get the sense that a lot of the expectations baked in are ones that are not just product level, but based on like misunderstandings of the future of AI and what it can do. So my guess is that because some people might be valuing like building God is basically infinite, there will be some amount of miscalibration there. And I think that, you know, like the let them cure cancer is going to be like Marie Antoinette's let them eat cake.
Jeb took the AI world by storm about a month ago. And I've been wanting to talk to Jogo, their CEO of TypeSafe, for quite some time. So it was really fun to have him on the podcast. I think maybe even better with a month to reflect on everything that's happened because we talked about a ton of different things. It was really awesome to get Jogo's perspective on the shapes of problems that he still thinks need to be solved by AI models, how he thinks about pushing JEV models to be even better. It was also fun to get his thoughts on Anthropics' upcoming IPO, AI safety, a bunch of other topics. I think folks will really enjoy hearing from him.
Without further ado, here's Jogo. Well, I've really been looking forward to this. I think ever since your viral launch, I feel like the second that happened, I was like, I have to have you on the podcast. And so I appreciate you making this happen.
Read the whole transcript
“what percentage of those calls going into the LLMs are actually meant for human consumption?”
play with it would be so helpful why'd folks tell you not to launch um we launched around the time of the pacing the frontier discussion and people were like hey like it's kind of like a bad vibe to like launch a capabilities product during this like everyone's like really doom and gloom about
ai and it's a kind of a bad vibe and like i don't know if you guys should do this and we're just like that discussion never really made sense in the first place um or it is at best incomplete in my opinion you talk about kind of you know wanting to push in in so many directions i mean i think one thing you know that you guys do is you're always kind of transparent about the things the models can't do, you know, areas where they're not as strong. How much of that is, like, the roadmap for what you're pushing on? Like, you know, how much of it is other areas? Like, you know, how do you think about, like, where you want the company to push in the next month or two?
Month or two? Oh, that's a... I mean, in AI world, I don't know if I can ask longer than that, right? I wouldn't say jaggedness is purely, like, our roadmap. There are some things that models truly are weak at. And, like, this is where, like, that taste stuff comes in, right? Like, when I was talking about like the reality of what you can extract from these models the reason we call them system one models and among other things and system one actually people did not want us to name that it either because system one thinking tends to be a bad thing like people use that as kind of like an insult right it's like the the part that just naturally happened from evolution you know from back in the day before uh exactly instinctually are much larger brains yeah exactly but and like people often like say that that is the mistake making side and i believe that the shape that actually ai gets compressed to happens to be like very system one and therefore the honest thing to do is to communicate its weaknesses because i think that on its own can be incredibly incredibly valuable for software and most of what people need It actually is system one stuff. Why can we solve Millennium Prize problems in math, but not automate anything touching accounting? That should be really weird to a lot of people. And I think that that is still our North Star, even if people get distracted by cost, speed, like, oh, it's good for really fast decisions or whatever else. I see these as distractions compared to what is really needed for AI to meet its potential, and that is my North Star.
And so obviously you put the models out there and you have a lot of people using them in different ways. How much does like the real world usage of the models come and feed back into the roadmap versus just like the stuff you all are planning to do? I would say it's almost none real world feedback. I think people should be complaining to us so much more. I think that like the field is still in a Stockholm syndrome of like, oh yeah, the model providers just do stuff in their one track path. And we happen to be like, the victim is the wrong word, but like um you know like in uh what's that movie wally you know like they're like they're just like the people in the chairs that are just getting fed the ai and i think this is like the state of things right now where like people plan around ai changing but they don't have a sense of agency of where to take the field and i think that this is not the way it should go that's actually really disempowering and kind of lame in my opinion and also i think that there's so much more we could do collaboratively as a field. And I think that the field could push us a lot more to say like, hey, we want to be doing these things. We think it is within the realm of capabilities. Let's
figure out how to do it together, but not just putting stuff into the model, but figure out how to prompt them, figure out how to unearth capabilities. Let's work together for these sci-fi use cases, because I think that is the way the future is going to go. And I would want so much more of that. My joke internally is that the number one way to get me to respond to something is to talk about model jaggedness because you have like very high up people say like, hey, can we meet with Yogo? And I'm just like, no, can you meet with my CTO or GTM head instead? But if they complain about model intelligence, I am like on that and like debugging that myself for better or for worse, because that I think is the thing that will really move the needle for us long-term.
So right now we mostly have to go with our old roadmap because we are not, I think people are still in that mode of like, Like, yes, let's just do whatever the model providers let us do instead of, like, let's collaborate on this, like, insane future together. And actually, there's another aspect of that. I think people have a deep distrust of model providers, too, because they train on their data. And, you know, like, this whole, like, discussion of, like, can GPT wrappers even exist, I think is very disempowering, too, because the whole application ends up becoming a chatbot. So the moat ends up becoming the equivalent of a system message that you send every time versus, like, the moat and the intelligence and the behavior and the specialization actually existing in software where we just make little micro decisions for you and you do all of the sick logic.
Yeah. So there's so much to do. There's so much to do. No, I think your point's really interesting because I feel like when Jeff came out, there became maybe like a common set of patterns or rules that folks were like, oh yeah, like this is a really good use case for Jeff and you should, you know, try it out in this thing or that thing and everyone tried it out and, you know, kind of got that sense of patterns. You know, I almost feel like people are trying to get their own mental models of like, okay, I use like, you know, Opus 5.5 for like this kind of thing. And I use Jeb for like this kind of thing. And I feel like your points, well, you know, your point is like, this is a dynamic evolving thing that you could, there's not strict boundaries, I guess, between those two types of model categories, but like, what's your ish, ish. I actually think that what you're describing is a little bit sad to me. And I, you know, I understand that like, we're playing a role in this. I do not think that that is the most developer centric worldview like for people to start building like theories of mind of the sub models they call and have their systems locked into that is i opinion, a really, really bad thing. I think the North Star for intelligence is going to be a lot
like SQL. Like, we want to be so reliable that it's actually provider agnostic. We obviously want to the best models, but like, we don't want it to be locked into like, things that only work with Jev. So what is your, you know, mental model for how this all evolves in terms of, you know, this like, thing people are doing today of like, you know, hey, you know, Jev works really well for these things. Maybe like, there's other sets of models that have been not like, to your point, And RLVR works really well for code and math. We'll see what else it ends up working really well for. I have to interrogate that gesture.
What was that about? Do you want to interrogate it right now? Yeah. Well, really well is quite relative. As far as I can tell, even the best companies still require humans in the loop for all of this stuff, despite superhuman results in some axes. and that still, like, people should have an answer to why that is the case. Like, if they are so unbelievably good at software, like, why do normal software engineers have anything to do? Not that I want them automated, but, like, they clearly aren't and there's clearly something missing there and I think thinking of, like, intelligence as this single axis that is measured by whatever the benchmark of the day is is, like, totally wrong and there's, like, obviously nuance missing there because the people who go all in on not understanding that nuance get burned super hard so i think that rlvr is incredible but he's incredible at human in the loop stuff still and i i think that wisdom comes from understanding reality where it's at knowing its weaknesses and like this is we are obviously use agents a lot but we really value understanding its limitations you know i just wouldn't say that it's like oh super great and it just solves everything personally maybe i'm using the wrong models
I think a lot of our listeners are trying to figure out, you know, like a simple mental model maybe of Jeb, which I know you disagree with is like, you know, hey, there's some set of tasks or some set of things I'm trying to do that will make sense for this. There'll be some set of things that make sense for other things. And I'm sure 95% of our listeners right now are trying to just like adjust their own mental model of like, OK, what am I like Jeb use cases? When do I use Jeb? that's a very pragmatic take for the short term and i'm in favor of it this is like where it's about giving engineers power because like we shouldn't be overly opinionated about what they use it on and if they need to do things to make it work abso-fucking-lutely hell yeah like if you need to do like the worst dirtiest hacks to provide value in the world like you know like we are not going to stop you uh like that is awesome and also if you have to do too many hacks that's still our fault you know like like in a deal where you wouldn't need to do that and the more it becomes close to like semantic understanding the more beautiful the code is the more reusable it is the more you can plug it in into places that you wouldn't expect it to it's kind of like
the unix philosophy of like doing like incredibly small useful things but because people have a mental model of it they can compose it together into big ones i feel like this is great and also like we should not be like recreating regexes we should be recreating like beautiful primitives not that i hate regexes i just don't like reading them you know and like you know like it's like It's like more, what's the word? Is it write-only code type stuff? And we want to be beautiful, composable. We want to build on layers like LLMs have failed to do so. And we want stuff to be more like software. I think I have an okay mental model of software engineers at worst.
It's kind of weird that people even want to update to the latest and greatest models. I hate updating my dependencies. Updating dependencies is the worst thing ever for production software. It's fun for personal use. I used to use Arch Linux. I get it. um and also like why would i want to bear the burden of that and the only reason people keep doing this treadmill of like maybe the next model will fix my problems is because i don't think they solve those problems so well that they can run in the background fuck off for literally ever and maybe cut you know revisit it every like couple of months to see like is there a cheaper or better alternative available but it works fine right now i don't i don't of anything like that.
And that is a sad state of the world. Like, it's very counter to how software engineering tends to work. I guess in your vision of the world, like, do LLMs have a place in the world in, like, 10 years? Super duper. Absolutely. I hope if that they do not have a place, then something strictly better exists. I'm not opposed to LLMs. I'm not actually opposed to either chatbots or coding agents. Completely different things. And I really don't want to go to a world where they don't exist. The thing is, I think people overuse them for a lot of tasks. And I actually hope the market for that increases too as people get better at it.
And I'm basically certain that there's tons of use cases for it at basically every single intelligence cutoff that has existed, maybe since GPT-4. All of these models had beautiful, awesome use cases that maybe would get Pareto dominated by another model, but they should be able to run in production. And it takes a lot of nuance and understanding. and it's like really not built for engineers. So I love LLMs. I hope that I can continue to like chatbot anything that I have like, I need like pseudo expert level answers to that don't need to be correct. That is the use case in my mind for chatbots.
And I love being able to like spin up, like throw away software instantly when I want custom dashboards and stuff because I think it's the best use case for coding agents. And that should stay for the rest of time. I don't think people should be doing that. and we should keep on building our toolbox, so to speak. Like, I think it's just awesome. Is your intuition that the toolbox will include, like, many other shapes of models in the future with, you know, other RL acronyms? Or if we're talking 10, 15 years from now, like, are there, you know, are there a bunch of different, like, tools in this toolbox?
Or is it kind of like your approach and the LLM approach cover most of the use cases? I'm going to end up saying something spicy here. but I would say when I first when we first launched this I was really excited for novel shapes to emerge um really really excited and I still think that that will be the case
“the shape really doesn't matter compared to actually it being useful and that is the thing”
Coding agents, like you said, very good for some shape of a problem. Obviously, folks have found JAV really helpful in different shapes of problems. Are there other problems out there that you're still like, God, there's just nothing out there today that's good for... So many. What are like some that come to mind that like of these future areas? Hi, I'm Anne Maris and I'm the producer of Unsupervised Learning. Thank you so much for listening to the show. I love working on this show and I have the challenge of hitting 35,000 subscribers by the end of the year.
If I don't hit that, I'll be fired. So please subscribe. Thank you. Bye.
“why can't we just have checks on top of every single tool call that actually are semantic to your organization”
that is the relationship i hope to have and this is why i'm like so so jazzed about software no i love it you said something there that i wanted to come back to because it was just a
“It's passing one giant dictionary around forever until the models break and they start getting dumber over time because context fraud is a real thing.”
“It's very measurable that multimodal models will be less reliable than the text models.”
like our image model is like maybe 10% of the way that our like text model is. As you move from like 10% by your characterization to like 90, 95%, what are the first kinds of things that you're excited about gem models doing on the image side on image land i have a soft spot for video games and computer use video games because i have like a video game background i and oh man i i confession i've also like played around with botting them not for competitive reasons but just because it's cool to automate stuff and i think like being able to play around with that and like i feel like software is a little bit like warping technology to your will and being able to like expand what you can warp to your will is just very cool in general so i think that is really cool in video games computer use i think is going to be super duper valuable unbelievably valuable i i think it's it's like especially combined with like lots of other stuff i think that ai should be so much better for interfaces it's actually kind of confusing to me how it is so bad you know like why there's a lot of people who don't like
using computers and they still have to even if they're like old and not experts despite that despite ai being probably better than an assistant they could hire to do a lot of computer stuff for them and i would love for that to be reliable enough that people could like trust it as an interface and you know i guess an additional thing that is like i have a soft spot for and this is why i'm wearing my like unsloppable shirt is i would love ways to just stop slop um for me personally it might be bad for the internet like the internet is funded by ads so like maybe it's a good thing i don't know but i would love if i could have some or someone would make me like a slop stopper where i just don't need to see it anymore that would be lovely but one thing i've been curious about from the outside is like the extent to which you know uh the advances that you've had at jev were enabled by like all the advances we've seen in in you know ai in the last two three years or could you have just done this five years ago like obviously you have a different approach that that is like in some ways orthogonal to the rest of the field like could you have done this five years ago or how much of what you built like does leverage some of the stuff that's been
done you know on the LM side? Five years ago, it would have been harder, but doable. Back when I was more naive, oh man, I sound like an old dude. When I was more naive and I thought that this would take a week of work, what convinced me, I love writing documents. I think it's a great way to think. I wrote a document at OpenAI that was titled, We Could Have Had AGI a Year Ago, at least by OpenAI's definition of AGI, which I don't think that people really listen to the letter of the definition, but I think it's actually a really great letter. I think that models from the starting from GPT 3.5 era, possibly even GPT 3 era, could have met the definition of AGI at the time, which is still the same, like doing most of the world's economically valuable work by volume. It doesn't say by volume, but I assume the spirit of that is by volume because the models are so unbelievably smart like gpt 3.5 level models surpassed median level intelligence at all sorts of things and like shouldn't that be used for tons of automation at least for the easy work and so yes i think that's something like this would have been harder but it could have been done five years ago absolutely and you know like on the people who are like obsessed with like for
example like the the benchmarks of the day and they see like what tasks ai is doing these tests are really hard. They're really, really, really hard. GPQA was saturated 2023-ish, maybe not totally saturated, but it was surpassing PhD student level at 2023, I believe. Google proof question answering for PhD level intelligence at these extremely hard multiple choice tasks, it was passing that level 2023. How is it that that level of intelligence was not being put to work that is super tragic so i think we've had a lot of like raw material for intelligence for a very long time and we're gonna see a whole lot of craziness yeah i mean speaking of that craziness i think part of i mentioned there's a lot of noise you know i thought there's been a lot of noise in the ecosystem and i bet many of our listeners will be familiar with you know after you guys came with these these models you had open ai and cloudflare microsoft and everyone kind of released these these models um i know you'll you'll say that the benchmarks aren't uh aren't a good way to test them.
They obviously have tried to show that they're near equivalent. I guess, how do you think about, like you obviously have now invited a bunch of competition into the space. How do you think about like what, you know, keeps Jev, you know, I guess everyone likes to ask about moats. So I guess I'll ask it on behalf of our listeners. Like, how do you think about that going forward for the business? Oh boy. It feels like other people are digging our moat for us, which is kind of not how I thought it would go. and I think that that's kind of bad for the field. I think that releasing models that are kind of more benchmark-optimized than real-world-optimized actually burns that goodwill for people to try things out, which is the only thing that matters.
I truly think that this is not great for the field. And my hope is that just giving unbelievable fucks about reliability is going to be the North Star. I think that will be immensely, immensely valuable for the world. there's just so much more to do there that I think that it's a little bit early to worry about competition. It's kind of like OpenAI worrying about a moat at GPT 3.5. Like, who cares? Like, GPT 3.5's market is nothing, or it was nothing, right, in comparison. It's still, I think, going to be nothing in the grand scheme of things. Like, we truly believe in Jevon's Paradox, obviously, hence the name Jev.
Like, Jevon's Paradox, like are we seeing rebound you know more demand because it's so cheap absolutely I believe every good sees rebound are we seeing backfire like the total market size increases this is this is when Jevons paradox holds Absolutely not. A key part of it has to be like, wow, this is so good and so cheap. I'm going to be throwing it all over the place because it makes value in all these places, and my budget for AI will actually increase. Speaking of investors, I guess you obviously recently raised a pretty big round. How did you even think about the amount of capital that made sense to raise for this business? I'm sure people were trying to throw money at you. How did you even think through that?
I will say that people probably don't believe it, but we can run our service profitably. And I hope that we never have to do the thing of temporarily subsidizing to get lock-in. I don't want to axe it out entirely because I'm a pragmatic person, but I really hope that doesn't have to be the case. I hope that... I find that to be a very shadowy, sneaky way to get people to use your service. So I and I hope to be the kind of person who's like, you know, always collaborating in like these prisoners dilemmas until, you know, until forced otherwise. So we actually did not need the money. And our goal instead was, hey, we want actually like the most valuable people on our side to help us build this business because we have no fucking clue how to do this.
So we really opted for like who are the best people in the eye who will help us. you can see who we got on our board well I always like to end with a quick fire round where I just stuff some overly broad questions into the end that I haven't had a chance to ask elsewhere I'm curious you know over the last month you know or last six weeks of having this product out there like what's one thing you've changed your mind on an AI in the last like six weeks having just seen how that's such a cool question whoa that's a quick fire question holy smokes I will say that we were really worried if our philosophy about caring about like the actual reliability of intelligence was a mistake like we had people in the team say like hey should we just release a bench max version and just hype it up and like do one of those early releases that like obviously is not great but just to like test out if the market exists i'm so glad we didn't do that and i i say seven weeks because like i i don't remember when launch was but like seven weeks i think was before launch we really did not know this we it was like a big leap of faith for us and like we were just really man like i don't think we could be like proud of ourselves if we did the same over promise
under deliver and like i think that that paid off super duper well i actually i don't know if it paid off maybe the product is sold itself and maybe it would have done even better if we like really optimized for for attention um but that was like awesome and i actually feel like now that my brand has been solidified as our brand I hopefully never have to do that like even close to that again and that debate no longer exists for us I hope I hope I don't know I truly truly hope because I have like a hard time like even giving like un-nuanced slightly incorrect answers because I think that that's just a disservice to anyone listening to me I would rather just like not answer the question than do the standard like CEO compression of the truth that errs on the side of generous so I hope that I will never really have to do that um but like something i've learned as well is that this reliability saga was sped run so fast for our class of models um i feel like i could be wrong but my temperature check of the environment now is people have already gotten like extreme fatigue of like the benchmarks and not actually useful stuff way faster than i would have thought like i i would have like thought that this would be a concern uh based on what i see because people keep falling
for benchmarks every time a new model is released right people are like well cool let's talk about it and then it's It disappoints, except when it's like an open-air anthropic model, apparently. And that just happened. I could be wrong, and maybe we'll see. I think it just happened a lot faster than I would have thought, because our model is so benchmarkable, such that people could blast out so many benchmark-crushing things so fast. I feel like this next one is gonna be very anticlimactic in comparison but I am as a podcast host I'm contractually obligated to ask you you know like we're we're an AI podcast the other big AI story right now is obviously anthropic um you know going public and folks kind of thinking through that business and I'm struck even in the course of this conversation like I actually don't know how you're going to answer this question because on the one hand I feel like you're very disappointed in like so many of the uh the lack of capabilities that LLMs have and maybe that they're like almost a hammer trying to like you know go on every nail of like different sorts of use cases. On the other hand, like you have been complimentary of like, there are things that coding
agents are really good for. And so I guess as you think about Anthropic is like, you know, one and a half, two trillion dollar company, like, is that something you're optimistic on is like,
“I actually get the sense that the greatest strength of the labs is not models or product or distribution, but financial engineering.”
that was that was a great one um i feel like the yeah i mean obviously so many people are underwriting you know the move into from coding agents to general knowledge work to bio to these other spaces and obviously like not the perfect use cases for rl vr by any means i still get the sense that the like the the nuanced way to understand agents perhaps and maybe all of ai
“my estimates are in the 20 to 40% speed up range. But again, it really depends on the use case.”
“i would guess that we are something like two breakthroughs of like gigantic magnitude away from this”
word to you, like anything you want to point our listeners to. I mean, I'm sure they're all familiar with. And certainly by the end of this podcast, certainly hope they are. But like anything we haven't discussed or I don't know, the floor is yours. There's so much stuff to have happened in software plus AI. I think it'll be so fucking incredible. I think that we need to start untraining ourselves of this lack of agency that LLMs have kind of taught us about AI and start really building again.
Like, there's, you know, like there's so many possibilities that are going to be opened up and we should get back to playing rather than being like these victims of stuff. And I will do whatever we can to make, you know, playable AI. But it's... I really want. Oh, man, I have like so much to say here. I feel like I will be doing whatever I can for the field to like have this opportunity to just rethink software, rethink interfaces, rethink everything. But it's going to require like real builders to figure out how to do that. And I think that together we can make AI this really awesome force for good in creation and just all around awesomeness.
you know, like, and sometimes even like really weird, wacky stuff. I feel like that's where technology really shines. I'm Jacob Effron, and this has been Unsupervised Learning, a podcast where I get to talk to the smartest people in AI and ask them tons of questions about what's happening with models and what it means for businesses in the world. As I hope is clear, I have a ton of fun doing this. It's a nights and weekends project, in addition to my day job as an investor at Redpoint. But our ability to get these incredible guests on really comes from folks like you subscribing to the podcast, sharing it with friends.
It's really what ultimately makes this whole thing work. And so please consider doing that. And thank you so much for your support and listening. We'll see you next episode.
