Optimizing AI Recommendations for Revenue with Sequen Founder Zoe Weil

Let's jump in. so anything that you see on the internet is ranked right whether it's a pair of pants that you're looking for at a retailer hotels that you're looking for for your x travel contents media even the ads you see online are powered by an auction and these seem maybe to most people to be unrelated but every dollar of online spend and consumption goes through a ranking model somewhere and we power those ranking models for the world's largest consumer companies walk me through kind of the the highlights of what you all are doing what makes your approach unique and you know with a an eye towards or ear towards the kind of technical innovations? Unsurprisingly, when people think about ranking, they think about information retrieval. In fact, I went to graduate school for information retrieval, which is literally the study of search systems, right? How do I retrieve the most relevant items or documents for a given query, whether that's in a search bar or some kind of relevance module somewhere out there?

And historically, and again, quite, I would say intuitively, the wisdom here has been that utility should be measured by whether or not I'm able to retrieve documents that are most semantically relevant to a user's query, right? That's a very technical side of it. And that works incredibly well. And I would say, by and large, maybe it's not a solved area of research, but there have been so many advances on it. It's been pretty remarkable. I mean, the frontier labs have it pat down or down pat. I forget what the idiom is, but they're pretty good at it. It's not really on its own an interesting, I would say, area of research.

however in a commercial context you know modern recommender systems if you think about tiktok instagram and the like they're not really recommending content anymore and they're definitely not trying to help people find things instead they're bending the end user's will in subtle ways over time to make them want things that they otherwise wouldn't want right so that destination vacation, you know, the latest fashion that you might not need, you know, whatever it is that you can think of that people end up just, you know, buying on Instagram or TikTok. They do that not because Instagram and TikTok are, you know, really good at helping you find what you're looking for. No, they're very, very good at influencing your behavior. And the reason that they're able to do that is that billions of people use their apps and are frankly addicted to those apps. They open their apps, you know, 10 to 21 times a day. And when billions of people do that, you have an insane amount of data on them. And so you're able to create these incredibly immersive, I would even say addictive experiences that are hyper-personalized and dynamic and in fact the apps react in real time to uh the user's feedback right so even you know

you sam if you what you see on the site or the app in the day throughout the day will vary based on your interactions with the app but the truth is that consumer companies regardless of how large they are, you know, regardless of how many customers they have. Wayfair is a good example. Priceline is another really good example. These are very large companies that have incredibly large user bases, do tens of billions of dollars of online transactions. But again, people don't use Wayfair the same way that they use Instagram or TikTok. And so regardless of how big you are as a consumer business, you know, you are going to be data starved. And so in an environment where you don't have as much data on the user, where people are not addicted to your app, how do you influence their behavior in a moment of high intent to increase their basket size, increase their average order value, to increase their engagement on your site? Again, these aren't like big changes so on wafer for example if if you're going to buy a hundred dollar couch it's not like sequin can come in and get you to buy a five hundred dollar couch and also a thousand dollar coffee table the point is As I can get people to buy $102, but I can do it consistently and at scale.

And when you do that, that translates to forexing their target profit for the year in less than a month. Upsell, cross-sell, like we've been, you know, doing this from a data science perspective, pre-data science. You know, we've been doing this for a long, long time, trying to get people to buy more. what's different about the approach the tldr is as i was saying earlier and in fact for the entire history of this field of research people have been trying the wisdom has been if i can make it easier for people to help to find things then by extension i can get them to convert better that has been the prevailing wisdom forever every company that's ever existed in this domain has been trying to solve search and relevance from the perspective of search and relevance, make it easier for people to find things faster.

That is not what Sequin does at all.

Zoe WeilWe are not at all trying to help people find things. We're trying to influence their behavior in a moment of time when they have high intent.

How do we do that? How do we go about doing that? Which is, I think, the meat of the matter. So we started the company with a black box solution that was powered by what we call large event models. So you can think of it as, first of all, it's not a large language model. We actually don't use large language models anywhere. That's not our business whatsoever. And we don't use generative AI either. And in any case, so we started with large event models. And the idea behind large event models is that the same way that large language models are able to generalize to text, including, you know, texts that they haven't seen before ever.

Large event models are able to generalize to human behavior and streams of events all in real time. And creating something like that is incredibly difficult to do for a number of reasons. One, you need to be able to create extremely ultra-low latency systems that are able to keep track of data at a very, very fine-grained level, right? So I'm not only talking about clicks and conversions and the traditional good stuff, although we use that as well, but I'm talking about hovers, scrolls. I'm actually talking even deeper than that. The speed of a scroll, right? The dwell time on a hover. How many times a user goes back and forth.

Creating systems that are able to digest data in, frankly, nanoseconds, transform this data also in nanoseconds, put it through a model, several layers of ranking models in milliseconds, and serve dynamic, re-ranked items to a user as they're browsing a site, all in under 50 milliseconds for thousands of requests per second, is incredibly difficult to do. In fact, we created what we refer to internally as sequin ranking clusters to be able to even make this feasible at all. And so there was a lot of work done there, not only from an algo perspective, but also from a data engineering perspective, feature engineering perspective, specifically for ranking, and a ton of MLops work and infrastructure work that was done.

It's hard because you need to do three things that are very, very difficult to do on their own. You need to have high-fidelity, leakage-free data pipelines, which is a really difficult problem on its own. You need to have... Obviously, models that are personalized to every user and continue to improve on those models. And then you need to be able to serve them fast enough, such that rankings are able to react to live user feedback and signals. So building this system at all requires you to do these three things really, really, really, really well. And there are very few people on the planet that can do this for the explicit objective of generating outcomes for businesses. Right? And then, so that's one part of the problem that sequence solves.

two even if you manage to build that system at all improving anything within that system is really really difficult because it's gated not only by how fast your scientists can do experiments and do research but the mlops component right the business component of it Like, how much am I trading off diversity or freshness of products or bookings or media content for direct impact to my bottom line, right? And so the ability to have this unified ranking interface that is focused and brings all these disparate parts of the organization together is incredibly helpful on its own. But the real secret sauce and where we started the business was with a collection of proprietary ranking algorithms and harnesses that are designed specifically, again, not to help people find things.

and I know I keep repeating this like I'm aware that I keep repeating this but that is really the golden thread because once you realize it and you see hey even though you're operating in the search space you're not trying to help people find things you can't unsee it you said that earlier and I think I had this like and I maybe expressed a little bit a degree of dissonance and that like that doesn't sound new to me and certainly you've lived this space more than I have but you know we've been talking about kind of next best offer and propensity to buy and like basket size optimization and all these things have been around forever.

And so is it like is it like old habits die hard and like you know they approach the folks approaching this in a traditional way approach it like a search problem and really the the key is to you know be like maniacally focused on business metrics which are kind of harder to do than you know focus on like traditional search metrics or or what yeah yeah yeah so actually the first part of it is is is true that you know there are really old habits very hard and they continue to see it as a search problem and try to solve it from a search perspective. Let me help people find things more easily. That is definitely a part of it. The other part of it is that like the biggest innovators in the space are social media companies and they publish a lot in this, right? Amazon used to publish quite a bit about this too, but they have a wealth of data that none of our customers have. So if you're a scientist at Wayfair or at Priceline or at any of the companies that we work with, you're going to read these papers and you're going to try to implement it. And you know what? Oftentimes you're successful at implementing it, except guess what?

It doesn't drive the same outcomes for you as it does for them. You don't have the data that they have. And Seguin's big unlock is that we're finally able to get you that unlock without you having that data. And is that kind of aggregating customers across? We don't. We don't. We do not share any customer data between our customers. And that is what makes us truly, truly innovative is that by relying so much on true in-session live signaling. I get it. So within the same customer interaction, you're extracting 10x, 100x more data than the typical. And so we've made ourselves very, very independent of that.

How do you respond to folks who kind of see that as like, you know, going deeper into this like surveillance capitalism kind of rabbit hole and like extracting more data about folks, you know, as they're trying to do pretty pedestrian things like, you know, that whole, I'm sure you've encountered that. Oh, yeah. I hate it. I hate it. I'm very, I'm one of the people that's probably the most vocal opponent. of that kind of, I love what you called it, surveillance capitalism. No, I absolutely hate it. And we don't do that. And for several reasons. One, and this is what distinguishes us from social

Zoe Weilmedia companies. We're not trying to create addictive experiences whatsoever. We're not trying to get people addicted to anything. We don't store people's information. It's all an

a live transaction and i i cannot tell you the number of times even i've been online um and i'm just trying to find well now i have a kid and i'm trying to find toys for my kid but before i try to find toys for my kid my my friend's kids and it's insanely difficult to do that right and i can actually i'm good at verbalizing what i'm looking for and it's still difficult to find that that thing that inspires you. And so what makes us different is that, you know, we're not surveilling people. We're not like chasing them from site to site to site. We're not doing any of that. You already have intent, which is why you're on Wayfair.com. You're on it already because you want to do something. We just make it easier for you to actually get to the thing that maybe you can't even verbalize to create moments of inspiration for you, spontaneous inspiration for you.

And so it does, yes, you know, Priceline ends up making a lot of money, Wayfair ends up making a lot of money, but the customers end up walking away happy. We're not doing anything like price gouging or like weird stuff. If anything, the difference is like maybe cents on the dollar per transaction that we're able to increase. In a sense, it's like you're using the data that's available through that interaction, like a salesperson might in a store. I noticed you were looking at that cell phone a little bit longer. I'm going to kind of steer you to things related to that. And maybe, maybe you actually, you know, you're looking at the cell phone.

What about this case also, you know? And, but we actually don't care about the identity of the user. We don't care about your name. We don't care about your last name. We don't care about your location. We don't really care about, like, you know, we really care about that interaction, the live interactive component. You mentioned MLOps several times. When you think about the kind of evolution of ML there and MLOps, like what are some of the big things that you and your team have learned in trying to put this, you know, platform or environment in place? I think we got lucky in a sense because maybe, like, six, seven years ago.

Rust didn't have a ton of support in, like, open source AI libraries like Hugging Face. That began to change rapidly in the past, I would say, three to four years. And, you know, this is why I always say whenever you make a successful startup, like, there's quite a bit of luck there as well. Timing is everything. Yeah, timing is everything. We got very, very lucky because we started building this company at a time when support for Rust and many, many open source frameworks, AI, ML frameworks. You think you could not have done it in Python?

Zoe WeilNo, I don't. I genuinely don't. I don't think that we could have because we, you know, ultra, ultra low latency feature transformations serving are so core to what we do.

I do not think we could have done it in Python. Certainly, we do a lot of R&D in Python, but the core of our backend is all in Rust. So no, I can definitively say we could not have done it in Python at all. And we are very, very lucky to benefit from a lot of the open source contributors that have, you know, for free, devoted their time and efforts to bringing support for languages like Rust. I'm imagining that another, you know, timing tailwind is the availability of coding agents now. You know, you may have needed to build a lot of, you know, supporting infrastructure before because it just wasn't available for what you're trying to do.

But now you can just ask an agent to build that for you. For sure. I mean, a thousand percent. You mentioned that Wayfair was an early adopter of transformers. Are your models transformer based? We have some that are transformer based. We have, I would say, all of our models certainly use self-attention, the self-attention mechanism. I can say that without a doubt. All of them do. We have many mixture of experts models now, but they all rely on self-attention. I wouldn't say that they use the transformer architecture across the board because, I mean, now we have so many proprietary models that we have built that kind of it's kind of like an orchestra of many many many many models it's like mixture of experts but for ranking is the idea at a system level that you've got all this data coming in and you are applying a bunch of different models is it routing or it's like you're simultaneously passing the same data through a bunch of different models to kind of come up with new features actually both right so there's definitely a routing component to it um because if you're like in the advertising space um things are going to be different for you than you know if you're just like in traditional commerce but

if you're if we're operating within an ads module uh let's just say wayfair uh there are things that the more specialized advertising models that we would use to power our, uh, uh, the bidding platforms, you know, might be helpful there. So, um, in that case, there's some bit of routing involved, uh, but also there's a simultaneous kind of information sharing and multitasking that we also do. The thing that I was trying to get at is a theme that comes up on the podcast quite a bit and it's this tension between like we're going to take a bunch of data and train some big mongo model end to end and it's gonna like take in everything and spit out the one answer versus we've got a bunch of data we're going to apply a bunch of models to it that you know in some ways are you know creating features and then those features are going to be used.

by some other model, you know, to, or, or in, you know, or some set of models or independently, it's like this kind of, you know, one big end to end thing versus, you know, we kind of know a lot about this problem. Like we're going to model, you know, individual aspects of the problem, the way we might think about it as data scientists. And, um, you know, that has, you know, some advantages around explainability, model size, et cetera, et cetera. Like We're closer to the second because we do have these mega models for like advertising domain or, you know, different domains, you know, but they tend to be on the whole much smaller than, you know, what you would see as large ginormous models in the industry because the mammoth models nowadays are so, so big.

Our largest models are a tiny, tiny, tiny fraction of that. What are some examples of like, you know, a model input, output transformation? Like, or I'm trying to get a sense for the granularity. I can give you a couple of examples here. So, you know, creating, even if you look at like creating embeddings, right, for queries, right? Very, very simple text. Queries tend to be smaller of size. Um, queries in different contexts and travel and, uh, and travel hospitality and, uh, media, um, and, um, in retail can mean completely different things. And so using kind of generalized language, these giant, giant language models to be able to create these holistic representations are not as useful to us in our context.

Because, again, keep in mind, we're a ranking business. We do one thing and we do it incredibly well, and that's re-ranking. So that's an example of where a lot of people would just use, you know, one kind of embedding model, a really, really great embedding model for text. that is able to contain the most amount of latent knowledge about intent, et cetera, et cetera. But like queries are like, what, one, two words, maybe? There is a lot, it's very information-starved. So our ability to narrow that to specific contexts and see just like two or three token queries in a very specific dimension within the user's overall journey because the user is the first class citizen at Sequin.

It's not, and by the way, when I say user, I don't mean the user identity. I mean the user is a collection of events in a session, right? When I think about a user, think about it as a collection of events in a session. I don't care if you're John Smith or Adam, whatever, to Sequin, you're just a collection of events and a live session and how I'm able to best represent you in, in, in that context is what sets me aside from every other provider out there. And, um, so we take these very, very raw streams of events, but where that gets routed to, in other words, how I end up representing that information to what then goes on to the final re-ranker is a big part of our secret sauce and what makes us really, really special. Is there a re-ranker or are there re-rankers for, I'm envisioning like, you know, we've talked about Wayfair. You go to Wayfair, you search for Blue Sofa, you've got your primary list and that needs to be re-ranking, re-ranked, but then there's like you may also be interested in this there's like a bunch of other stuff on the page that we power all of it the way yes and there are different they're they're very different so i'll give you an example even a very tangible example that i think you'll be able to that any uh listener technical or less

technical will be able to follow uh think about like complementary products right so let's say you bought a tennis racket um and uh i'm not going to show you other tennis rackets you just added it to your cart, right? In fact, that will most likely dry down conversion. Well, you would think, but then we also have the classic, I just bought a refrigerator on Amazon and now they're showing me refrigerators. Like, why is that? And that's why you would use sequin because we would never do that. That's exactly, no, I'm serious. But anyway, you go on the add to cart page and we show you complimentary items.

So we don't show you other tennis rackets. We show you, hey, what about this like sweatband or this tennis ball? But which tennis band and which sweatband or ball we show you is going to vary from user to user. But the point I'm trying to get at is that that complementary module exists throughout many of these sites. So it'll exist on a product display page, but it'll also exist on an app to cart page.

Zoe Weiland how you rank items on a product display page, even though it's still a complimentary module, will vary greatly from how you rank items

on a complimentary module on an Add to Cart page. And that's because by the time the user gets to the Add to Cart page, they've already committed to the tennis racket, right? Whereas I need to be able to show greater diversity on the product display page because they haven't necessarily committed to that tennis racket yet. Does that make sense? Yes and no, right? You have greater diversity in... That's just one example, but the point is that they're both complementary modules. Right, but I'm saying like if I search for a tennis racket, I mostly want to see tennis rackets. I don't want diversity in that list.

You don't, but once you click on it, you click on a specific tennis racket. Right, then you don't know what I'm going to need next. Exactly. Well, we're betting that we have an idea before you do or we can get you to click and purchase. Well, so this is an interesting thread. Like how, so I'm, you know, I'm in this journey. I search for tennis racket. A bunch of tennis rackets have come up. I choose a tennis racket. I add it to my cart. And then you show me a bunch of other stuff. How does any of the information you've collected about dwell time and all these other things while I'm looking for tennis rackets apply to you know which headband which set of balls which you know anything else like a ton of things oh my god you give you give the site so much information you might not even realize so like how how did you end up deciding between the blue tennis racket let's say and the pink tennis racket how long does it take you to decide that and how does that compare to every other sequence of events that every other user has ever had like is it classic like uh collaborative filtering you know folks who no because in this case keep in mind the identity of the user actually doesn't matter right and it's all in real time right so i look at the sequence of events that i've

collected in real time when people usually let's just say uh hover for x amount of time go back and forth x number of times before settling on this tennis track in a live session an implicit personality modeling like this is a uh overly analytical anally retentive person he keeps flipping back and forth between these two things he's scrolling over these lists kind of you know he's like scrolling over the specs or something like that yeah so that he He probably wants, you know, something different from. the person who, like, is flipping through all the pictures? No, it's the difficult thing to do, again, more than anything else, is, like, keep in mind, the data, the environments you operate in are incredibly data-starved, right?

So with as little information as possible, what can I do? And most of the reliance comes from streaming in-session events and getting all of that to a model. First of all, the data comes raw in a live session, So you have to clean up all that data in literally nanoseconds. And it has to go into a model, which has to then spit it out in milliseconds. So it's a lot when you think about the entire journey that the raw data has to go through. But yeah, the idea is based on how you got to finally adding that pink or blue or whatever color tennis racket to your cart, I'll show you just the tennis ball.

or a sweatband, uh, or bag that you're going to, you're going to buy. And in fact, one of the, one of the things that we are very, very, very good at is, um, increasing basket size, like regularly. Um, the other thing that we're really, really good at is typically historically, um, you know, uh, as revenue and organic revenue are considered competing metrics, which makes sense typically usually when ad revenue goes up it goes at the expense of organic because you at the end of the day have a list of items and either organic goes to the top or odds

Zoe Weilor whatever and what we've been able to consistently show is that we're actually able to drive both up together um so that one doesn't cannibalize the other which allows us to compound these

massive massive gains on incredibly mature relevant stacks elaborate on that for me when you talk about ad revenue are you talking about ad spend that's getting the user to the site or are you talking about like at an amazon where there's promoted site promoted items versus second okay right so i guess i think about that from an amazon perspective because it's you know very you know it's it's surfaced it's you know more or less transparent you know a little bit more than less you know whereas i don't know that when i think about wayfair i think about sponsored items versus non-spider sponsored items but you know a side of that scale is probably doing that whether or not i know about it is kind of the i think they're fairly transparent about it i just think that you see it more on amazon because amazon has a much much larger ads business and they're a huge huge company and so you're saying that you know typically when when an organization is kind of uh they optimize ad revenue as a as the primary objective conversion tends to go down yeah yeah okay because people will because they're optimizing for people clicking on stuff as opposed to buying things as opposed to buying things and what sequin

has proven that we can do time and time again is we can get people to click significantly more and get them to also buy after the fact why is that is the same thing we're talking about just more data like is it that you're individually optimizing these things and so they both go up or is it some cross no they're they both go up we're just creating like much more we're showing people things that they actually want to see um based on what we call micro feedback that we get things like scroll speed and stuff yeah it's funny you you you say that because i was thinking just a bit ago about how you know typically the the big kind of aha moment for relevance is if you can find And, you know, some feedback, like any feedback, and usually it's like, or, you know, historically has been in, I'm thinking about like RAG systems, for example, like, you know, did the user click?

Like, you know, that's a, did the user click on the thing that we showed them? Oh, like, or did the user thumb up or thumb down? Like, you know, there's, I would say teams have kind of been starred for like, you know, any kind of feedback and like they you know have very few levers for this explicit feedback and now like you're just creating way more implicit way more feedback implicit feedback yeah yeah no and we rely i would say overwhelmingly on implicit feedback data overwhelmingly it's crazy i would say i was talking to i was talking to and i'm not going to name because you know i don't want to embarrass them but i was talking to c-level executives from like a very very major bank so they're like you know you'd be lucky to work with us we're this giant bank and we have all this transaction data and blah blah blah i was like actually that's like that's the least interesting data you could give me because you're only showing me a sliver of human behavior that transaction data is useless because 99 of people don't actually convert i'm interested in the 99 nine percent of data you actually don't have and that's what i have and you don't have that we didn't get very far in stack like we mentioned russ like i'm i'm the things that come to mind as

you describe the system or questions are like analytical databases no sql you know queues confluence kind of thing like what are some of the infrastructural the data infrastructure elements that allow you to do this yeah so we definitely use a lot of no sequel uh for sure um uh distributed databases we definitely use rest uh you rely heavily on um we rely very very heavily on open source uh technology um we use a lot of uh hugging face huge fans of hugging face here. Um, I can tell you also what we don't use much of, uh, we almost do not use LLMs at all. Um, it's just not relevant to our business. Uh, we're not trying to do generative anything. We're not a lab. Uh, we're not trying to reinvent the wheel there. Um, uh, so yeah, honestly, it's just a, it's a lot of ranking models. It's actually like the thing is that getting these models to work requires you to have an innate understanding of what features drive value. And that is not written in any research paper. And so what we have done is we've created a, uh, harnessed library with proprietary, uh, institutional knowledge that we've codified, what we call residual knowledge, um, that meta has never published about, uh, that Google has

never published about, that Pinterest has never published about, their biggest moneymakers. Even the papers you see them publish, they're not. The scientists that work on them have to wait for months to get legal approval. And the version of those papers that actually ends up getting published is a very, very watered down version of that line of work. And so what we've done is that we have created a harness library and a knowledge base of some of the most effective proprietary methodologies to drive outcomes in ranking. And some of it has- come from hiring folks from these institutions and, you know, gathering residual knowledge that's not published anywhere. And then second of all, we have four deployed engineers embedded in all the customers we work with. So we actually don't need feedback data from the customers, but residual information that we gather about what works, what doesn't work is all codified.

You mentioned or you referred to, is the harness and recursive like the same concept or is the harness a subconcept or? Harness is one component of the entirety of our recursive stack. So you can think of the recursive stack as like being this engine with many, many different components. We have a proprietary knowledge drawer. We have a harness library. We have what we call frontier ranking models, all of which work together to power the recursive stack. And so our core IP is the recursive stack itself that produces these insane ranking models that are able to, you know, 4, 5, 10x Fortune 500 companies revenue in just a matter of weeks.

and that's why we are able to allow them to then own the ranking models right because the real meat of the matter is the ability to continuously build ranking models better and better and better and better ranking models that continue to drive revenue and so when you talk about this harness what makes it a harness is it just a library and like harness is the cool word or is there something harness-y about it in the same way that, you know, Codex or Quadcode is a harness? Yeah, so it's actually a collection of harnesses that we have. Definitely not, certainly not a cool word at the time that we started building it in.

We just needed a way to specifically have what we call eval harnesses. to find what we call, uh, the commercial ground truth of interactions. And so that's why we started to build these harnesses and we quickly realized in short order that, uh, even, you know, in the, in the, in the space of evaluating these ranking models, uh, how we were able to do so offline in, let's say, the travel space, which is, you know, considered obviously higher ticket items, has a ton of overlap with the furniture space, for example, which also has higher ticket items. But guess what? It doesn't have a lot of overlap with the ad space because the user is just clicking on things and that translates directly to revenue, but it's not coming at a cost to a user so how you evaluate these different kinds of ranking models um you know requires to you to have a very very rigorous approach and so we came up with a collection of um these evaluation harnesses and that's that was the start of our of our harness library you call the the tool or the platform recursive like what's recursive about it recursive ranking intelligence is kind of the first name.

Is that, is recursive just a name or is recursion a part of what you're doing? Actually, if I were to be very, very specific about it, it's most similar to like TL recursion, meaning that you're able to,

Zoe Weilinstead of having to rebuild the entire ranking, iterate on the entire ranking algorithm and rebuild it from scratch,

which I would consider iterative and not recursive, you don't have to rebuild the entire ranking stack every time um you're learning so you're able to learn continuously and make uh minor modifications here and there and in doing so you're able to compound um gains at a much faster rate at a significantly less at a fraction of the compute so that is if i were to get very technical that is why we call it recursion is that we don't have to rebuild um ranking models um previously when we started the business um we actually did so we would train the ranking models a daily sometimes hourly but it would be like from scratch and it took us quite a bit of time actually to figure out a way how can i have how can i um quote unquote iterate on this without having to rebuild it from scratch every single either day or hour or whatever cadence and and insert that recursive element in there.

We talked about this a little bit earlier, but kind of with the context of the full conversation thus far, what is large event models? Is that just an instance of one of your ranking models or is that a particular architecture or? It's one kind. It started as the thing that we did. Let me be very clear. that in itself was insanely difficult for us to do to create these models that were able to generalize to stream of events it sounds extremely intuitive and it is but getting it to generalize across customers and domains is insanely difficult like intuitively it makes a lot of sense but to get it to work in travel and media and retail at a different cross points different geographies without relying on any end user identity is really, really, really difficult to do.

So that was one of the hardest things that we began to work on. What's the input? Is it a stream of events? It's just a stream of events. And each event kind of has a label and a value, for example? Yeah, so it would be like hover combinations, right? So it would be the query, the embedding for the query, and that query could either be literally the source item or it could be an actual query of words that a user searched for or whatever. That was the first, very, very first thing that we worked on and it was the black box model on top of which we built rankers.

Zoe WeilThen we built what we called large mesh models, which were like large event models, but they're much more relationally focused.

So now I'm looking not only at streams, but i'm looking at many streams simultaneously and looking at the relationship between those streams together and trying to create generalized representations out of that right um then we started creating different classes of what we called frontier ranking models which were independent of large event models and large mesh models but which large mesh models and large event models contributed to, right? And so we built these collection of many, many, many now in the hundreds of thousands of frontier ranking models that we have that are proprietary, that are not published anywhere. A lot of them with very novel architectures. And that then became the first building block of the recursive stack. Over time, we started to develop harnesses and knowledge store, et cetera, et cetera. To what degree does the idea of time series models or time series modeling play into what you're doing? Do you think of these events as time series or do you just kind of capture a frame of events?

kind of process it uh independent of you know longer time sequences earlier i will tell you that earlier it played a much much bigger role um earlier in our journey now i would say it's a small part of a growing stack um and i'll tell you why um i get i think i can give an example that is perhaps intuitive for people to catch on to. You know, time series and sequence-based modeling works really, really well for languages and generalized reasoning because you can show A follows, sorry, B follows from A. And there is a general order to things. And even though we're in the ordering business and all our whole business is ranking, I'll take you back to the fundamental concept that their first class citizen at Sequin is the user and the user does not behave as cleanly or logically as you would expect. There's a lot of randomness in people's behavior. People aren't like languages. Language, despite however creative that you can get, has a certain repetition to it that human beings simply do not have, right?

You could be in the middle of buying a couch, right? And your dog just comes barking in and completely ruins your train of thought. And then you suddenly realize, oh, I'm going to buy a dog bed too. You know, there is something inherently unpredictable about human behavior. To the degree that we've talked about, like a big unlock being like capturing more information from the user. Capturing the events at the browser is certainly complex, but doable. But then you talked about like the infrastructure to manage those events at scale. It's a lot. And then transform all of those in real time. And then know what to transform it into.

Yes. Which, by the way, we have dedicated models to do that. So you'll literally have streaming data that's going in not to one model, but it's going into like hundreds of models. which all have to then output that data, reconstruct it, put it into the final ranker, and have it all be served to the user in sub-20, sub-50, sub-100 milliseconds. It's an enormous infrastructure and modeling undertaking. With that in mind, when you think about kind of the technical horizon or the innovation opportunities, like what are the big opportunities to go after that will help you kind of push your system and approach further?

There are several routes that we're looking at right now without revealing too much. I think, you know, one of the biggest debates we've had internally is how much do we invest in traditional representation learning of images or texts and the like versus actually continuing to innovate on the ranking models. themselves and the nuances there. And I was talking about how our eval harnesses and the library there is absolutely critical to our success. And so I would say right now, at this

Zoe Weilpoint in time, anyway, we're very, very focused, I would say on the latter, rather than focusing on things like that we think are going to be very commoditized, if not already, you know,

in terms of having multimodal representations of like hero images on company websites or whatnot. So I would say that is probably where we're most focused. on are on things that actually make the ranking, the core ranking model itself better rather than the auxiliary components that are, that involve more traditional representation learning. We're going to leave that to the big image or video or other frontier labs to do. And open source projects will continue to benefit from that research as it, as it comes out. Well, Zoe, it's been great chatting with you and learning a bit about, you know, what you're up to and how you're approaching ranking uh it's been super interesting uh thank you so much for having me uh it's been a it's been a huge pleasure and i'm a huge fan of your podcast so awesome thank you it's an honor to be here Thank you.