A good tutor keeps each lesson just beyond the student's reach

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 1:59:16

   With respect to Eureka, I think one thing that is very fascinating to me about education is, I do think education will pretty fundamentally change with AIs on the side. And I think it has to be rewired and changed to some extent. I still think that we're pretty early. I think there's going to be a lot of people who are going to try to do the obvious things, which is like, oh, have an LLM and ask it questions and do all the basic things that you would do via prompting right now.

2. 1:59:38

   I think it's helpful, but it still feels to me a bit slop, like slop. I'd like to do it properly and I think the capability is not there for what I would want. What I'd want is like an actual tutor experience. Maybe a prominent example in my mind is I was recently learning Korean, so language learning. And I went through a phase where I was learning Korean by myself on the internet. I went through a phase where I was actually part of a small class in Korea, taking Korean with a bunch of other people, which was really funny.

3. 2:00:06

   But we had a teacher and like 10 people or so taking Korean. And then I switched to a one-on-one tutor. And I guess what was fascinating to me is I think I had a really good tutor. But I mean, just thinking through like what this tutor was doing for me and how incredible that experience was and how high the bar is for like what I actually want to build eventually. Because I mean, she was extremely, so she instantly from a very short conversation understood like where I am as a student, what I know and don't know.

4. 2:00:34

   and she was able to like probe exactly like the kinds of questions or things to understand my world model. No LLM will do that for you, 100% right now, not even close, right? But a tutor will do that if they're good. Once she understands, she actually like, really served me all the things that I needed at my current sliver of capability. I need to be always appropriately challenged. I can't be faced with something too hard or too trivial.

5. 2:00:57

   And a tutor is really good at serving you just the right stuff. And so basically, I felt like I was the only constraint to learning, like my own. I was the only constraint. I was always given the perfect information. I'm the only constraint. And I felt good because I'm the only impediment that exists. It's not that I can't find knowledge or that it's not properly explained or et cetera. Like it's just my ability to memorize and so on.

6. 2:01:15

   And this is what I want for people. How do you automate that? So a very good question about the current capability, you don't. But I do think that with, and that's why I think it's not actually the right time to actually build this kind of an AI tutor. I still think it's a useful product and lots of people will build it. But I still feel like the bar is so high and the capability is not there. But I mean, even today, I would say Chachapiti is an extremely valuable educational product.

7. 2:01:44

   but I think for me it was so fascinating to see how high the bar is and when I was with her I almost felt like there's no way I can build this. But you are building it, right? Anyone who's had a really good tutor is like how are you going to build this? So I guess I'm waiting for that capability. I do think that in a lot of ways in the industry for example I did some AI consulting for computer vision.

8. 2:02:06

   A lot of my times the value that I brought to the company was telling them not to use AI. It wasn't like I was the AI expert and they described the problem and I said don't use AI. This was my value add. And I feel like it's the same in education right now, where I kind of feel like, for what I have in mind, it's not yet the time, but the time will come. But for now, I'm building something that looks maybe a bit more conventional, that has a physical and digital component and so on.

9. 2:02:29

   But I think there's obvious, there's obvious, it's obvious how this should look like in the future. Do they assume you're willing to say it? What is the thing you hope will be released this year or next year? Well, so I'm building the first course, and I want to have a really, really good course. state-of-the-art, obvious state-of-the-art destination you go to learn AI in this case, because that's just what I'm familiar with, so I think it's a really good first product to get to be really good.

10. 2:02:53

    And so that's what I'm building, and NanoChat, which you briefly mentioned, is a capstone project of LLM101N, which is a class that I'm building. So, that's a really big piece of it, but now I have to build out a lot of the intermediates, and then I have to actually, like, hire a small team of, you know, TAs and so on, and actually, like, build the entire course. And maybe one more thing that I would say is, like, many times when people think about education they think about sort of like the more what i would say is like kind of a softer component of like diffusing knowledge or like um but i actually have something very hard and technical in mind and so in my mind education is kind of like the very difficult technical like uh process of building ramps to knowledge so in my mind nano chat is a ramp to knowledge because it's a very simple it's like the super simplified full stack

11. 2:03:36

    thing if you give this artifact to someone and they like look through it they're learning a ton of stuff yeah and so uh it's giving you a lot of what i call eurekas per second yeah which is like understanding per second that's what i want lots of eurekas per second um and so to me this is a technical problem of how do we build these ramps to knowledge and uh so i almost think of eureka as almost like a it's not like maybe that different maybe through to some of the frontier labs or some of the work that's going to be going on because i want to figure out how to build these frontier these ramps very efficiently so that people are never stuck um and everything is always not too hard or not too trivial, and you have just the right material to actually progress.

12. 2:04:15

    Yeah, so you're imagining the short term that instead of a tutor being able to probe your understanding, if you have enough self-awareness to be able to probe yourself, you're never going to be stuck. You can find the right answer between talking to the TA or talking to an LL and looking at the reference implementation. It sounds like automation or AI is actually not as significant. And like so far, it's actually the big alpha here is your ability to explain AI codified in the source material of the class.

13. 2:04:48

    right? That's like fundamentally what the course is. I mean, I think you always have to be calibrated to what the capability, what capability exists in the industry. And I think a lot of people are going to pursue like, oh, just ask ChatGPT, et cetera. But I think like right now, for example, if you go to ChatGPT and you say, oh, teach me AI, there's no way. I mean, it's going to give you some slop, right? Like when I, AI is never going to write NanoChat right now, but NanoChat is a really useful, I think, intermediate point. So I still, I'm collaborating with AI to create all this material. So AI is still fundamentally very helpful.

14. 2:05:17

    Earlier on, I built a CS-231N at Stanford, which was one of the earlier, actually, sorry, I think it was the first deep learning class at Stanford, which became very popular. And the difference in building out 231N and LL101N now is quite stark because I feel really empowered by the LLMs as they exist right now, but I'm very much in the loop. So they're helping me build a little materials. I go much faster.

15. 2:05:40

    They're doing a lot of the boring stuff, et cetera. So I feel like I'm developing the course much faster and those LLMs infused in it, But it's not yet at a place where I can creatively create the content. I'm still there to do that. So I think the trickiness is always calibrating yourself to what exists. And so when you imagine what is available through Eureka in a couple of years,