Teach by exposing the simplest useful model first

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 2:15:26

   Can I ask some questions about teaching well? if you had to like sort of like give advice to another educator in another field that you're curious about to make the kinds of YouTube tutorials you've made. Maybe it might be especially interesting to talk about domains where you can't just like, you can't test somebody's technical understanding by having them code something up or something. What advice would you give them?

2. 2:15:48

   So I think that's a pretty broad topic. I do feel like there's basically, I almost feel like there are 10, 20 tips and tricks that I kind of semi-consciously probably do. But I guess like on a high level, I always try to, I think a lot of this comes from my physics background. I really, really did enjoy my physics background. I have a whole rant on I think how everyone should learn physics in early school education because I think early school education is not about crumbling knowledge or memory for tasks later in the industry.

3. 2:16:16

   It's about booting up a brain. And I think physics uniquely boots up the brain the best because some of the things that they get you to do in your brain during physics is extremely valuable later. The idea of building models and abstractions and understanding that there's a first order of approximation that describes most of the system, but then there's a second order, third order, first order terms that may or may not be present.

4. 2:16:36

   And the idea that you're observing like a very noisy system, but actually there's like these fundamental frequencies that you can abstract away. Like when a physicist walks into the class and they say, assume there's a spherical cow and dot, dot, dot. laughs at that, but actually it's brilliant. It's brilliant thinking that's very journalizable across the industry because, yeah, cows can be approximated as a sphere, I guess, in a bunch of ways. There's a really good book, for example, Scale. It's basically from a physicist talking about biology. And maybe this is also a book I would recommend reading, but you can actually get a lot of really interesting approximations and chart scaling laws of animals. And you can look at their heartbeats and things like that, and they actually line up and with the size of the animal

5. 2:17:15

   and things like that. You can talk about an animal as a volume and you can actually derive a lot of, you can talk about the heat dissipation off that because your heat dissipation grows as the surface area, which is growing as square, but your heat creation or generation is growing as a cube. And so I just feel like physicists have all the right cognitive tools to approach problem solving in the world.

6. 2:17:35

   So I think because of that training, I always try to find the first order terms or the second order terms of everything. When I'm observing a system or thing, I have a tangle of a web of ideas or knowledge in my world, in my mind. And I'm trying to find what is the thing that actually matters? What is the first order component? How can I simplify it? How can I have a simple thing that actually shows that thing, right?

7. 2:17:53

   That shows an action. And then I can tack on the other terms. Maybe an example from one of my repos that I think illustrates it well is called micrograd. I don't know if you're familiar with this, but so micrograd is 100 lines of code that shows backpropagation. It can, you can create neural networks out of simple operations like plus and times, et cetera, Lego blocks of neural networks. and you build up a computational graph and you do a forward pass and a backward pass to get the gradients.

8. 2:18:18

   Now, this is at the heart of all neural network learning. So micrograd is a 100 lines of pre-interpretable Python code and it can do forward and backward arbitrary neural networks, but not efficiently. So micrograd, these 100 lines of Python are everything you need to understand how neural networks train. Everything else is just efficiency. Everything else is efficiency. And there's a huge amount of work to do efficiency.

9. 2:18:39

   You know, you need your tensors, you lay them out, you stride them, you make sure your kernels orchestrating memory movement correctly, et cetera. It's all just efficiency, roughly speaking. But the core intellectual sort of piece of neural network training is micrograds, 100 lines, you can easily understand it. You're chaining, it's a recursive application of chain rule to draw the gradient, which allows you to optimize any arbitrary differential function.

10. 2:18:57

    So it's a, I love finding these like, you know, the smaller terms and serving them on a platter and discovering them. And I feel like education is like the most intellectually interesting thing because you have a tangle of understanding and you're trying to lay it out in a way that creates a ramp where everything only depends on the thing before it. And I find that this, like, you know, untangling of knowledge is just so intellectually interesting as a cognitive task.

11. 2:19:24

    And so I love doing it personally, but I just have fascination with trying to lay things out in a certain way, and maybe that helps me. It also just makes the learning experience so much more motivated. Your tutorial on the Transformer begins with bigrams,