A simpler search system worked as language models improved

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 10:38

   I actually read this Paul Graham tweet I think like where if you try to often when you when you figure out the better solution when you try to solve a harder version of it but you end up with a simpler solution that's more general and scalable that's what we realized like okay like there's one way of doing these things where we go to each of these domains and like try to build an index of it and put it into specific formats like tables and then have the LLM like read that in a structured language like SQL.

2. 11:10

   Or you could do the other way where you just keep it unstructured and expect the LLM to do most of the work at the inference time, at the time of the query. And don't do all this work in the indexing time. And clearly we knew that if second is where the world is headed, where the models will get smarter and smarter, it gives you an advantage to build it that way because it's more general. and uh you also stand a chance against the legacy system that google has built which is a lot more in the first style so we thought okay we would try to build a more general solution and then we prototyped this thing one weekend actually actually john chilman's team had already published this thing called web gpt at the time so i was pretty aware of it opening i even had a bot when i worked They're called the TruthBot, which John built with this.

3. 12:00

   where you could ask it a question and go and search the web and then it'll give you an answer with some sources. And it was very slow and it was built with the 175B GP3 model. So incredibly slow and inefficient. It was more agentic, like it would actually be like an RL agent that decides if it wants to click on a link and browse it, scroll. It was very slow. So what we tried is a very simple heuristic version but much faster which is okay you always take the top k links that a search api provides you you always only take the summary snippets that the index is already cached so there's no scrolling there's no clicking and you always feed all those links into the prompt so there's no selection ask it to write a summary with sources in like the academic format and that's it when these models were getting to a point like 3.5 turbo sort of models were

4. 12:55

   beginning to come, this actually started working much better. It got really good, yeah. Instruction following capability increased enough that you didn't have to do it very, very rigorously. Gotcha, so you kind of did like the dumb approach, betting on the fact that the AI would get good enough that it would make all of that irrelevant. Right timing, I would say. One year ago when John and his team tried, the models were just so much worse that like if you tried the dumb approach, it just wouldn't work.

5. 13:26

   And so therefore you would conclude that you need a smarter approach. But then when the models began to be much better instruction following, the dumb approach actually works. And that fixes a core product UX problem of latency.