Agents need search built for different inputs and latency

Transcript

From the transcript Knuckle Up with Nakul publishes with the episode.

1. 39:04

   So let's double click on it. Parallel web is search for agents. Why do agents need a different web search? What's different about how agents search than humans? A lot, right?

2. 39:19

   First, let's take the top-down perspective. If agents are going to use the web 1000x more, can you build 1000x the search infra that Google has built to support? I bet no. Okay.

3. 39:33

   So now you have to have an argument that you can do something novel and unique. Okay. So you ask the question, what will this thing look like? And here's what you have. You're seeing the best capability

4. 39:51

   to show up at the hardware layer in GPUs. You want a lot of memory. You want lots of memory bandwidth. You want flops. Get GPUs. What if you decided to frame the web search problem as a problem to go solve it primarily using GPUs over the generations of commodity GPUs. Second, these models are getting better. The other thing that's happening is holding capability constant, you're going to be able to make the models smaller and smaller and smaller.

5. 40:24

   And so now you will be able to Distill into a much smaller model a lot of intelligence in a very broad-waved way to build your ranker.

6. 40:37

   So this is the dimension which is new hardware and technology advancements as it relates to AI is going to allow us to build something which you couldn't have built a year ago or two years ago. That's number one. Number two, Agents don't behave like humans on the web. So if you think about it, let's talk from a few dimensions. Inputs. A human going to a search engine, their input is a short, underspecified search query.

7. 41:15

   You're sometimes even using autocomplete, right? So mostly we don't tell the search engine what we want because we are impatient. And we haven't even thought about it, right? Like you're trying to do something and you like write two queries and then you go on this random walk of clicking on links, reading paragraphs, going back, searching more. That is how humans work. Agents are happy to write a session's worth as a query, right?

8. 41:44

   For human search engine, the output is 10 blue links so that you can go on your random walk, right? For an agent, The output is a collection of, let's call them tokens, to put in the context window

9. 42:00

   that give the agent its next set of action space, correct? Which isn't just clicking on links, right? Number three, think of this human search engine. It monetizes at a certain rate. It has a certain latency profile to it.

10. 42:21

    And those two things, Roughly tell you what kind of compute to throw at the problem and what kind of architecture to use, right? So it monetizes better than I think agentic traffic will because agentic traffic will be a thousand X. Two, the latency profile on humans is humans are like a page loading between 500 milliseconds. It's great, right? It takes two seconds. It's really slow and you'll be annoyed.

11. 42:51

    If it takes 400 milliseconds or 500 milliseconds, you won't even notice, right? So there's a sweet spot you hit in page loads in search. With agents, there's high variance here, right? Think about a voice agent. If you take 500 milliseconds, 600 milliseconds for a voice agent, It feels too long a pause before the agents can talk back to you if you just do the math. Also, the entire premise of agents is humans essentially have a one-second SLA in life, right?

12. 43:21

    Like we want things to be interactive on the web, loading a page, talking, voice agent coming back in sub-second. Agent's entire promise is you're going to be able to do more within each turn, right? So you ideally want search. In an interactive setting, to go from this sort of 500 millisecond thing to a 50, 100, 200 millisecond thing, depending on the use case. But in the background agent context, search can take an hour. Who cares?

13. 43:52

    It's working overnight, right? So now you have this extreme variance in what could be useful. And this open canvas... To build entirely new technology around entirely new optimization points, the input is different, the output is different, your compute budget is different, your latency budget is different. What we've solved historically is this one narrow slice of it.