Recursive reports beating community results on AI benchmarks

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 29:24

   You've introduced recursive a bit in our conversation thus far, but we haven't like formally dug into what you're trying to do and, you know, where you are today with it. You know, talk a little bit about guys publish a blog post on kind of your first steps towards automating AI research. I guess the first thing is that like you've got this broad vision of, you know, accelerating science, but your aperture for recursive is AI research in particular.

2. 29:54

   That's where we want to start. We feel like it's helpful to start there first, get the AI to be really, really good, and then eventually apply it to all kinds of innovations. What are some of the early things you're seeing in some of your early experiments within the AI domain? Yeah, so one thing that we applied it to and we published a blog post about is this thing called NanoChat, where you basically automatically do automate research people call it auto research basically this is Andre Carpathie's auto research project that's right yeah so that is like a baby version of what this kind of Eureka machine can do and so we applied it to that and two days later like it had outperformed by far everything else that everyone else had kind of collaborated on and worked on in this space and and what's the what's the structure of that

3. 30:55

   benchmark like what's the the task the idea is that yeah the task is you have a model and you try to get to the lowest bits per byte within five minutes um on like a particular piece like of hardware or gpu um and so you know that has the advantage of like well now it's just very quick you can have this like fast inner loop so that makes everything a whole lot easier and so um Um, we, we basically got the lowest bits per byte, um, compared to anyone else, like by a significant margin.

4. 31:25

   Um, and then, uh, we said, well, it's nice if we could, um, also be faster. So there's another one called NanoGPT speed run. Um, it's somewhat similar, but, um, it's harder to beat the state of the art because just a large community has been optimizing, uh, solutions for it for almost two years now. so instead of asking just like how low can you go within a fixed time budget you say i will have this fixed validation loss and i want you to get there as fast as possible and so there again people have been working on it for two years and After a couple of days, we outperformed what everyone in the world had worked on together with their agents, some of their auto research and their own ingenuity.

5. 32:09

   And then we said, OK, these are all cool, but let's get closer to things that really matter for the world. So we applied it to the sole exec bench, which is a NVIDIA benchmark that was supposed to be unhackable and you couldn't reward engineer it. We found some ways to still do that. But again, within a couple of days, like we got, have the best kernels across multiple different examples, better than cursor and double AI and everyone else in the world on almost all the kernels.

6. 32:43

   There may be like half a dozen of the 200 odd kernels that we didn't have the best version of. And yeah, that was really, really exciting. And that's just like, you know, one part of the inference sort of system that you can optimize with our technology.