How To Build The Future: Aravind Srinivas

At the same time, it's very grounded in product building. It's very difficult to do that. And I spent a lot of time thinking about it. I even spoke to Ilya Sutsukiwa about it where we said there are probably only two problems where you can work on AI and also build product at the same time. One is like search and the other is sub-driving car because all your product rollouts are becoming data points or improving the underlying AI in the product. And that'll make the product even better. And that'll lead to more users and more usage will lead to more data points and it'll become a flywheel. And it should also be on the AI completeness path.

It's sort of a buzzword to say this, but basically what it means is better AI should keep making your products better. So that way you can keep working on your company until AI is solved. And once it's solved, okay, sure, we'll worry about all those implications. But your company gets better as AI gets better, as opposed to your company gets run over by somebody else. Exactly. So search is like one of those problems. So you're at this moment where you kind of have this realization that you want to start a company. How did you get the kind of activation energy to quit your great job at OpenAI and go do that?

How did you find your co-founders? I came across this blog that one of the... Former YC partners Daniel Gross wrote, but it was like, how to build the next Google. And I think basically the core idea is like, you could do so much more with better query reformulation. So you take a query and you just add some suffixes. So if someone's looking for reviews of a movie, just suffix side colon rottentomatoes.com. If someone's looking for reviews of some new gadget, do side colon that corresponding subreddit. You can get away with a lot of these suffixes and like our special strings to like filter results and already make Google so much better, even with the existing Google ranking.

I'm not even talking about the ads problem, just simple ranking. And then you can do more sophisticated things of like classifying queries.

Aravind SrinivasAnd he was talking about how LLMs could automatically figure out these suffixes. And I was pretty interested in that.

Okay, that seemed like, okay, maybe generative AI might be, as in, like, I used to just call it LLM, so generative models could be a better way to build search engines, too. I also was pretty interested in, like, trying to do something agent-like when DeepMind had this Android environment that they built where, like, they kind of wanted to prototype a mobile app using agent that knows when to use what apps and control the apps. that's when I spoke to my co-founder and CTO his name is Dennis we had written the same paper a day apart so we knew each other and he was a visiting student in my lab and we used to talk about I bring some ideas of how like we can build agents to control the Android environment so we were definitely chatting about doing a lot of things but never concretely about any company or product the first thing that anyone would tell you is like why why even work on this of course Google's going to do this right it's not even like you go build a better google docs uh and google will eventually do it because it's a secondary thing for them so companies like notion can still be funded this is their core crown jewel so why would you even try i think the reason it actually made

sense this is again after launching the product we realized this not before um so there's like some benefit to being ignorant, ignorance is bliss, is that if people stop clicking on links, the ad economy kind of dies. Now you can, there's a lot of nuance to this, but that core insight was only realized by us after launching. So once we realized that, I thought, okay, we were onto something and that kind of took us last two years. Walk us through like the first iterations of your experimentation, like I know you did a bunch of demos that were very dissimilar from perplexity. Yeah, so I was like adoshes enough to go and pitch to the first seed investor of ours, Elad Gill, that like, hey, like, you know, I want to disrupt Google, but I kind of want to do it from pixels from a glass.

And I think that's why, you know, you're not competing with people typing on the search bar. They're just seeing. But even at that point, you like knew in your mind, I want to go after Google or like. Yeah, it was not like I want to go and kill Google, like that sort of motivation. It was more like, what is an idea of that scale and ambition is something like this. It was also around the time when multimodal models were slowly beginning to work. So I thought, like, if you were on the trajectory of improving technology, you could build something pretty amazing. My investor rightfully said, like, not to work on it in the beginning.

So we focused more on searching over, like, specific verticals or data sets or databases, tables, actually. And we were an enterprise kind of focused company, except, like, nobody wanted to give us their data. I remember I used to, you know, hustle for calls with like PitchBook or Crunchbase because I kind of wanted to build a demo that would first make sense to an investor. That way we can keep, you know, raising some capital and then actually hiring good people and then go and like do the real thing. And so Crunchbase had all this data, PitchBook has all this. They just don't want to give it to us.

And so... Next best step, Twitter. Yeah, Twitter. The pre-Elon CEO moments, academic access was allowed, legal. So we built a database of Twitter. We organized it in the form of tables. We tried to do it with the OpenAI codex models. This was even pre-GPT 3.5. We wrote a lot of templates. Oh, for these kind of queries, these are example sequels. And then the model would do kind of rag, they would pull relevant queries in the templates and then write the actual SQL based on the template sequels. And that was the only way to get it to work reliably.

Aravind SrinivasAnd then we had a lot of callbacks in case errors happen. It'll automatically correct it.

And then it will go and query the database and then retrieve the records. It was very nice. And it was a chat UI. You could chat, you could converse, you could plot. And this was like the first real product or demo that you guys did? Yeah, we did it very fast. took only a month to do this because there were like three people only but that energy in the beginning was insane and they showed it to a bunch of people and they all allowed it there are two reasons one, something like that never existed before like you could never actually Twitter searches exactly to this day even today, right? and then also people allowed finding all these I think the social search of like knowing who other people are following whose tweets are they liking Whose tweets are they not liking?

Who did they unfollow this week? You know, those kind of diffs. It's all like funny. So you launched this Twitter search thing. Yeah. How did you transition from that to now what we all know as perplexity? Yeah. So we had that, right? And then we were trying to do something similar to that for many different databases, GitHub. Like if coders could go and search about repos or LinkedIn, if you could like almost be a recruiter or just say, but even now it's pretty hard. I want to say, like, I want all the people who worked, who have been YC founders, you know, and who also worked at a COD startup, because they would know what it means to be scrappy.

It's very difficult to, like, do these queries. Oh, yeah, using the LinkedIn UI to do that. You cannot do that, right? Yeah. For whatever reasons, people don't want to give their data, their paywall, like, you know, such technology, if it exists, we'll be creating, like, way more value. But it doesn't exist for many other reasons. We were beginning to see how, even with the capability of the models at that time in 2022, pre-3.5 Turbo, things were actually pretty reliable to the extent where people would use this and find value. I actually read this Paul Graham tweet I think like where if you try to often when you when you figure out the better solution when you try to solve a harder version of it but you end up with a simpler solution that's more general and scalable that's what we realized like okay like there's one way of doing these things where we go to each of these domains and like try to build an index of it and put it into specific formats like tables and then have the LLM like read that in a structured language like SQL.

Or you could do the other way

Aravind Srinivaswhere you just keep it unstructured and expect the LLM to do most of the work at the inference time,

at the time of the query. And don't do all this work in the indexing time. And clearly we knew that if second is where the world is headed, where the models will get smarter and smarter, it gives you an advantage to build it that way because it's more general. and uh you also stand a chance against the legacy system that google has built which is a lot more in the first style so we thought okay we would try to build a more general solution and then we prototyped this thing one weekend actually actually john chilman's team had already published this thing called web gpt at the time so i was pretty aware of it opening i even had a bot when i worked They're called the TruthBot, which John built with this.

where you could ask it a question and go and search the web and then it'll give you an answer with some sources. And it was very slow and it was built with the 175B GP3 model. So incredibly slow and inefficient. It was more agentic, like it would actually be like an RL agent that decides if it wants to click on a link and browse it, scroll. It was very slow. So what we tried is a very simple heuristic version but much faster which is okay you always take the top k links that a search api provides you you always only take the summary snippets that the index is already cached so there's no scrolling there's no clicking and you always feed all those links into the prompt so there's no selection ask it to write a summary with sources in like the academic format and that's it when these models were getting to a point like 3.5 turbo sort of models were beginning to come, this actually started working much better.

It got really good, yeah. Instruction following capability increased enough that you didn't have to do it very, very rigorously. Gotcha, so you kind of did like the dumb approach, betting on the fact that the AI would get good enough that it would make all of that irrelevant. Right timing, I would say. One year ago when John and his team tried, the models were just so much worse that like if you tried the dumb approach, it just wouldn't work. And so therefore you would conclude that you need a smarter approach. But then when the models began to be much better instruction following, the dumb approach actually works.

And that fixes a core product UX problem of latency. You are used to like links appearing instantly on a traditional search, right? Even then, by the way, the first version we launched, which is the answer version, took seven seconds or something to, because we didn't even have this concept of streaming answers we would wait till the entire answer was generated we couldn't control the verbosity so sometimes the answer would be very very big we even had to hard code a prompt saying only write five sentences or something like that or 80 words keep it fast yeah exactly okay so you launched this when was the first moment that you thought like oh I'm on to something here so we tweeted it okay I was while writing the tweet, I was like, you know, people are going to ridicule it. It's going to make mistakes, blah, blah, blah. First moment of virality came when one annoyed intellectual academic came, searched for herself. It said she, it gave a biography in the past tense.

And she's like, I'm still alive. What the hell? But actually what happened was there was a person with the exact same name and spelling who died. And LLM thought she died and she gave a past tense. I actually thought that was pretty clever reasoning on the model, except it's not even higher order to know that they're different people. So then that got us a lot of attention. People were beginning to start thinking, okay, look, the sources thing is good, but can we really trust the answers these things are saying? And then that got into this trend of people searching for themselves. This is something that keeps happening time and again with all consumer products. When I got a chance to speak to Mike Krieger that vacation, he said the same, that even though you can click on your own profile icon and go back to your photos, people always love to go to their profile on Instagram by typing their username on the search bar. It's such a human habit. So a lot of people start putting their Twitter handles or social usernames, and then it would mash all their activity across the internet including stuff they did in the childhood like many years ago and then give like those interesting summaries and they would

screenshot it and share it yeah so i thought there was something there something driving it yeah that you but i still wasn't sure yeah and then we released the ability to ask follow-up questions. That doubled the engagement time on the site and also increased the number of questions every day. And the number of people, the number of questions every day was increasing exponentially. So I was like, okay, this is, there's something here. It's not worth killing and pivoting to enterprise. You have this like initial momentum. And you said earlier, it wasn't until hindsight that you had the idea that like, oh, we actually have a chance of competing with somebody like a Google.

When did that realization happen in this journey? How'd that go down? So I never really thought about the Google competition in a serious way, to be very honest, because I knew that, like, they cannot make this exact product on the Google homepage. It's so hard to know when a query is purely informational or not. And then the Google search page is already, like, so cluttered. There's the answer box, the knowledge panel. There's some ads. There's some links. There's, you know, like, perspectives from socials, all these social cards. it's already too much information. So it clearly feels like fast food and healthy meal difference for using Google and perplexity on even informational queries.

Aravind SrinivasI was more worried about Microsoft in the beginning because they were launching Bing Chat.

In fact, on the day, we agreed to have a term sheet, like handshook on a term sheet with one of the venture capital investors, NEA, here in Sandwich Road. after like one week of torturous pitches and we were just having like coffee and then the words leaked screenshots of Bing chat and I was like, okay, like there's this 30-day due diligence period, right? And one of the other investors would give me a term sheet. He just increased it to 45 days. You know, you could see the diff. Yeah, right. It was in sneakily and I knew why clearly. He also texted, what do you think about this thing? Okay, okay.

I get it. I get it. getting a little sheepish and then the other person i handshook with like he texted me the night saying hey do you have time for a call tomorrow okay like clearly like this is it right so i told my co-founder okay maybe they're gonna back out or ask us to pivot so maybe we should just try to sell the company and like get it done you know this is not gonna go anywhere the person actually who handshook said look i'm not gonna ask you to pivot i'm not gonna ask you to like do anything different uh you guys keep going and uh we already word is word i was like damn that's that's pretty impressive and then the next week actually google also releases a blog from sundar saying they're announcing something called the bard which is screenshots so we knew that like this is gonna get pretty big and competitive but we were like look it's at the end um microsoft was never really good at consumer products for a long long time you can't suddenly changed that uh so they actually messed up the opportunity in my opinion totally google obviously i knew that they're going to have their own problems and challenges so i felt like there was space for someone else here yeah having spent almost a decade at google myself um i see a lot

of the culture of the early days of google like the things i've learned about larry or about sundar and i see a lot of that in the way that you have built your product like there's a lot of attention to detail feels like you are the primary user of the product yourself like is that a thing that you deliberately tried to do yeah i did i did deliberately try to do it uh one thing that larry says like you know we we uh i keep reminding everyone in our company about it the user is never wrong so even today while testing a new feature um it didn't work uh but there was some ambiguity in the query so the person i was talking to the engineer and saying hey you know this is not good. What else could AI have done here? And you know what the AI should have done? It should have come and clarified to me and asked me, hey, I'm not sure either this or this. Which one did you actually want? And then I should have clarified.

and then it should have gone and done it. Instead of saying, I don't know, that is the user is never wrong principle. The other way of designing products is like, make the user be a better prompt engineer. Blame the user and tell them to be a better prompt engineer. Teach them, educate them, get them to do it the way that the product wants them to do it. Yeah, exactly.

Aravind SrinivasEnterprise software is more like second kind, but magical consumer products are more the first kind.

I agree. Like in Google, why should Google handle typos? They needn't have, right? We should have all been great at English. It's like Larry says he was never good at spellings, and that's why. I think the true story is YC partner Paul Bukai, he was just annoyed by it. And he's like, someone should build that. Yeah, exactly. And spell check, corrector, it's all there. Similarly, auto suggest. Why is it there? Like easier, right? Similarly, cached results. I was even reading somewhere where Larry wanted the homepage to have the simulation of the weather outside your home so that you don't even need to type the weather query.

It's just already there. So I was very influenced by that style of design, like including like subtle things, the Chrome search bar, if you've already gone to a site, it's already there. You just have to click enter after typing the first two letters. So that influenced me to like, make sure we have the cursor ready to type on the search bar. You don't need to take your mouse and place it there. It sounds like your main metric that you care about is number of queries per day. Which is exactly what Google did, I think in the early days, right? It's hard to grow that without like retention. In the long run, I agree.

You cannot just pay for a user and get that number up. User could install your app and maybe you can even game it where when they install, that's one query automatically submitted, but a repeat query doesn't need to be submitted. Yeah, I think the only counterexample, which I don't think is happening in your case, is the product is not serving their needs. And so they need to issue a bunch of queries to get what they want, which is kind of the opposite of Larry's approach on Google was you should be on Google as short as possible because we're trying to get you somewhere else to solve the problem. So that's not happening. I mean, sure, I'm sure there are some errors and stuff, but most of the follow-up queries actually we see are like completely irrelevant to the first query because they just want to keep continuing the session or questions that they never even knew they wanted to ask, but they want to keep asking.

So I presume your team has grown a bunch of money. How do you manage the team? How do you operate your team on a week to week or cycle to cycle basis with that, you know, number of queries per day is our primary metric. So every all hands, we start with that number. Okay. I don't believe in this, um, putting a TV and having a metric, you know, being seen every day, because I think that's also distracting, but I do think like it makes sense to take a look every week, see the weekly growth rates, um, see the monthly growth rates and like if something declined and discuss about it figure out ways to actually freak out if something declines like we do and something grows like look into why where so we are very data driven and we share it across the company actually i've been trying to share to the users too so that they feel like uh you know it's an something that's actually happening right in front of their eyes and they want to be part of it there's a hierarchy like if if there's some bug to be fixed if i know some particular person's

Aravind Srinivasworking on it i can go and talk to the person directly nobody else feels threatened because i'm going and talking to that person there is no feeling that because i'm raising a bug uh uh it's

like oh they're going to be fired or something uh because i raised like 50 bugs a day so if you know like it's more like they understand okay this it's important for the product to feel uh great and if it doesn't feel good for ourselves then the user is also not going to feel that in fact we have way more incentive to go use our own product, but the user doesn't. So the standards for the user should be even higher. So always feel like a user. I think that culture is there in the company. I love that. And did you intentionally select for that when you were hiring? Like people who are very product centric and in the details?

I wouldn't say I explicitly have that as a criterion, but I look for people who cared about doing good work. If you don't care and you're just treating it as a job, then it's very hard for you to get excited about things. And I think so much of it feeds off of the founders and like your culture, your DNA. And it sounds like you're that type of person that obsesses over the details. And you're just going to naturally want to hire people who share that trait. Yeah, I do get pissed off if answers are wrong, and I do get pissed off if people on Twitter are saying, like, perplexity is degrading or, like, you know, but a lot of the things, some things are actually not true.

But I do try to see, you know, leave aside the cynicism, even if it was someone who's, like, a hater. But if there was something true there, and I want to still know. Yeah, I love seeing you engage on Twitter with customers. is that the primary way that you talk to users or are there lots of other ways that you are talking so i mainly use x twitter people are just like super like brutally honest there and uh i think in email people a lot more polite yeah which is okay too like i like both sides but i think the brutal honesty brings out the worst bugs and uh things that people are afraid to say oh and in person is the worst where you go show someone something and they're just going to tell you good things yeah even if they hate it i kind of like don't like any hey what hey tell me what do you think yeah you're always going to say nice things right you're gonna grow your company presumably you're going to need to hire more people how do you avoid the fate of becoming a big slow company well it's uh beginning to happen already a little bit right we're not as fast as we used to be i think some of it is not because of people it's also because things breaking in production people start losing trust in the product like today we deployed some change and

then someone got frustrated that there was some front end bug somewhere it was actually something back end but people are just assuming things I think like fast loading all that stuff matters and not every new engineer has the full context of the code base in their head the earlier ones too there's some tension in like moving fast and breaking things if you do want to like grow to mass market usage uh so that's mainly slowing us down i would say uh and i haven't we haven't quite figured out like the best way to do this fast i mean we do have staging deployment testing a b test and all this stuff's happening and that's naturally slowing us down like getting things out to production widely other than that i would say uh the obsessive detail-oriented people So there are only that many people in the world.

So obviously you cannot expect engineer number 250 to be like that. But I try my best to still like, you know, go flag bugs to whoever's working on whatever new feature and kind of like know who's working on what, even at the size. I still try. I think our co-founders are amazing. They care and they push that principle when they're like building their teams. So we are trying our best. I'm not saying we've figured out or cracked it, but at least we're trying to fight the entropy here. I think that's the only thing you can try, right? Yeah. And it's an uphill battle, but if you keep on it, yeah.

Okay, let's talk a little bit about the future. I've seen your most recent launches are kind of like in different directions, more verticalized or more specific around shopping or other things. Where do you want to take it? Like today, my view of perplexity is a more intelligent Google search that's really useful in certain scenarios. What do you want me to think of it in three or four years? If you go and research what's the best sweater to buy or which is the best hotel to stay in this location, perplexity will give you a great answer. But where do you actually go? You go and fulfill the demand. You go to Google. And who gets credit for that monetarily?

Google us. Google. We got nothing. Maybe we get you a pro subscription, but then someone else will undercut us and give it away for free with cheaper models or whatever. They have bigger cash results. So the challenge is you want to be one place where people can have the end-to-end experience. they start with a problem in their mind and they seek your help you give them the answers and you also help them fulfill the action it's difficult because uh people think like at the app if you have an answer of like oh like what watch does uh bezos wear i think he wears some omega something um i personally thought it would be amazing if it not only give the answer but it also had a product card for the specific omega watch with a buy button and i just click buy and it's done but there are other people in the world who think that's an ad it's not even an ad right they think like that company is paying us to do this so this is where like some of the tension with uh the early adopters who love the ad free informational experience with like what is actually needed to get mass market and really be useful on a daily basis comes from there are so many other things like checking the score of a game or quickly getting to a website uh if you just wanted

to get a docs link of an api or if you just wanted to go and book a flight on united the answer could just be a link the answer could be a the weather for tomorrow that's the temperature or someone's age should be like you're going to type like elon musk network you'll just get the answer in like less than a second on google right and perplexity it'll pull the right source maybe it might be more accurate than google but people don't care about like some of those minor details so what you need to build is this amazing orchestration of small models uh typical knowledge graphs uh widgets lm streaming answers and more complicated multi-step reasoning answers but user doesn't care like user's not going to tell you like when to use what you decide that ai nobody talks about like when to use WOD, that sort of router, that orchestrator.

I think that's the hardest thing to build.

Aravind SrinivasAnd whoever builds that and can operate that at a scale of a billion users and also knows how to monetize some of those queries really well

is going to be the next Google. Because they'll have the search bar. They know exactly what to do. They'll go and ask clarifying questions. It truly understands the user and also does tasks for you. and also lets you like surf the web in a typical way, all in one experience. You could even argue maybe nobody will ever be able to build this because it feels like a daunting task. But I could say whatever Google has already built is the closest system to something like this. Agreed. So the next generation of this clearly can be built. You just have to like persevere and work for a decade or two on this problem.

If I talk to people at Google, they would say, yep, that's what we're building. In fact, I know they've been saying that for more than a decade. Probably the same at OpenAI, probably the same at Anthropic. When you look at the people that you likely will be competing against in the next 10 years, what do you think is the piece that's maybe going to give you the edge to win? Obsession about the user and good product taste. There's a lot of these things that require a lot of domain knowledge. Out of the list you mentioned, Google is the only company that actually has the product taste to do this.

and arguably all the distribution in the world, everything, right? Except the dilemma is also there. It's funnily like, you know, it's a search company, but it's also an ads company. And search is kind of almost exists in service of the ads company, not in the other way. And you could argue, okay, that's outside the search revenue every quarter, which is like close to... $200 billion a year, there's still like $100 billion other places, YouTube and cloud. But the margins are all coming in search, right? Cloud is only like recently profitable. YouTube is not, never going to be a high margin business because number one, they don't serve ads on subscription users.

And number two, they have to pay the creators, they have to pay the media partners. So it's never going to be as high margins as search. So you're arguing basically the stock price is going to be their encumbrance. Correct.

Aravind SrinivasBecause Wall Street is like crazy. It just automatically panics if search revenue goes down.

But search revenue has to go down in a world where people are just directly talking to AIs and agents are doing stuff for them. That doesn't mean they're not going to do anything about it. They're still building Gemini and the new app. The hypothesis is that they're not going to be able to easily put it on the core Google where they already have all the billion users. And that's true, right? Yeah, you're arguing that whoever wins this in the long run will kind of by definition need to come up with a new monetization model, a new business model. Yeah, there's like a ton of other problems to solve like for shopping or travel or like all these things like which merchants do you use or like which hotels do you plug into or who's the middleman there and who handles the booking and if a customer wants to cancel stuff.

Google actually solved a lot of these problems too, right? They're not just like, oh, a page rank or like MapReduce or, you know, all these advances that they made in like visual, like deep learning and like BERT, Transformers. It's not just that. That is great, but they also did a lot of other boring work of bringing Google Finance, Google Shopping, Google Flights. I feel like Perplexity is a better position to do these things than OpenAIR Anthropic because we have it in our DNA to care about the user and the product. we're not just talking about reasoning and models right? But we are pretty much familiar with all those things and we are very much capable of taking the latest open source models and serving them ourselves and fine-tuning them and post-training them and doing evals we're not like AI illiterate. We're not going to like spend all our bandwidth building data centers and chips and like trying to just talk about like breaking the most reason coding a math benchmark I think there's value in that but it's quite orthogonal to like building the next generation information experience all right arvin thanks so much for joining us it's great chatting thank you for having me again