The Future of Inference Market & Model Routing: Commodity or Increasing in Value? | LangChain CEO

Summary · 3 min read

Harrison Chase is the cofounder and CEO of LangChain, which builds developer tools for creating, testing and deploying AI agents. LangChain has announced approximately $160 million in funding, works with 35% of the Fortune 500 and has surpassed one billion open-source downloads; LangSmith processes over one billion events daily.

Chase says agents remain difficult to make reliable because non-deterministic models create large input and output spaces, with errors that can compound across tools and model calls. He argues that agent builders can create durable value through domain-specific context, harnesses, memory and evaluation, and that vertical agents often need routing built into their harness. He sees memory and continual learning as unresolved problems, requiring systems that turn agent traces into measured improvements. Chase says demand for inference and coding agents has exceeded his expectations, while rising token use makes cost controls increasingly important.

Minute by minute

1. 3:04

   Better models erase product features. Chase says LangChain sometimes has to remove parts of its product as models improve and make earlier workarounds unnecessary. He distinguishes short-term hacks, such as parsing generated JSON for tool calls, from durable needs such as connecting models to proprietary data and APIs.

2. 5:31

   Neutral harnesses preserve portability. Chase says agents build up state, including memory and context, that users should not have to tie to one model provider. He argues that a controllable harness helps users switch models and retain control over context, data and tools, especially for uses outside the coding tasks that labs often emphasize.

3. 8:34

   Domain expertise shapes vertical agents. Chase says domain experts can create value by defining what good performance means and building verifiers and evals. He advises starting a vertical agent with context and harness changes, then considering model post-training once it has users. He says post-training already suits simpler tasks such as classification and extraction, while its value for the core agent loop remains an open question.

4. 13:48

   Routing depends on the harness. Chase favors routing inside the harness for vertical agents because it can use more state, including prompt-cache information and task-specific prompting. A gateway makes routing easier to use with different harnesses and may suit general chat or coding, but he says domain-specific routing needs to understand the task and harness. He does not consider routing a commodity.

5. 19:52

   Evals and traces guide improvement. Chase says public academic evals can be less informative when models are optimized for them, while domain-specific evals remain useful if teams also inspect traces and data. He sees value in using those traces to improve an agent’s context, harness or model, then measuring whether the change helped. He describes continual learning as an outer loop that repeats this process.

6. 33:24

   Agents raise infrastructure and token costs. Stebbings raises an example where an agent checking for tickets every five minutes could generate 1,000 times the requests of a human; Chase says agent traffic increases strain and requires systems that can scale more broadly. He says frontier models are getting cheaper, but agent use is driving token costs up. LangChain spends roughly 30% of engineering salary on coding tools, and uses visibility, budgets and harness changes to manage spend.

7. 44:27

   Open models suit narrower tasks. Chase says some enterprises worry about Chinese open models, but considers those concerns largely unfounded when models are hosted by a provider or internally; he says evals can help address concerns about model behavior. He expects comparable US open models within 12 months. Today, he sees open and post-trained models used more often for routing, guardrails, extraction and summarization than for core agent loops.

8. 47:52

   Inference demand exceeds expectations. Chase says he underestimated demand for both inference and coding agents, and expects demand to support multiple providers, including Fireworks, AWS and Crusoe. He would invest in inference, lower-level infrastructure and vertical agents built on deep domain expertise. He also looks for strong technical talent, fast execution and brand.

9. 54:18

   First movers still need execution. Chase says being first can build a durable brand, but points to AutoGPT as a reminder that early popularity does not guarantee continued success. He describes Benchmark as a close company-building partner and Sequoia as helpful with recruiting and launches, and says a valuation discount for top-tier firms can be worthwhile. He dislikes fundraising posturing; in the rapid-fire round, he says agents remain hard to build and chooses OpenAI over Anthropic at the proposed valuations.

Published Oct 17, 2026.