LLMs do not build organizational context on their own

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 14:55

   And the simulator, in this case, happens to be an autoregressive transformer. So a transformer is a neural net. And it just kind of like goes on the level of tokens. It goes chunk, chunk, chunk, chunk, chunk. And there's an almost equal amount of compute for every single chunk. And this simulator, of course, is just... It's basically there's some weights involved and we fit it to all of the text that we have on the internet and so on.

2. 15:21

   And you end up with this kind of a simulator. And because it is trained on humans, it's got this emergent psychology that is human-like. So the first thing you'll notice is, of course, LLMs have encyclopedic knowledge and memory. And they can remember lots of things, a lot more than any single individual human can because they read so many things. It actually kind of reminds me of this movie, Rain Man, which I actually really recommend people watch.

3. 15:43

   It's an amazing movie. I love this movie. and Dustin Hoffman here is an autistic savant who has almost perfect memory. So he can read like a phone book and remember all of the names and phone numbers and I kind of feel like LLMs are kind of like very similar. They can remember SHA hashes and lots of different kinds of things very, very easily. So they certainly have superpowers in some respects, but they also have a bunch of, I would say, cognitive deficits. So they hallucinate quite a bit, and they kind of make up stuff and don't have a very good sort of internal model of self-knowledge, not sufficient at least, and this has gotten better, but not perfect. They display jagged intelligence, so they're going to be superhuman in some problem-solving domains, and then they're going to make mistakes that basically no human

4. 16:27

   will make, like, you know, they will insist that 9.11 is greater than 9.9, or that there are two R's in strawberry, these are some famous examples, but basically there are rough edges that you can trip on, so that's kind of I think also kind of unique. They also kind of suffer from enterograde amnesia, so, and I think I'm alluding to the fact that if you have a co-worker who joins your organization, this co-worker will over time learn your organization, and they will understand and gain like a huge amount of context on the organization, and they go home and they sleep, and they consolidate knowledge, and they develop expertise over time.

5. 17:02

   LLMs don't natively do this, and this is not something that has really been solved in the R&D of LLMs, I think. And so context windows are really kind of like working memory, and you have to sort of program the working memory quite directly because they don't just kind of like get smarter by default. And I think a lot of people get tripped up by the analogies in this way. In popular culture, I recommend people watch these two movies, Memento and 51st States.

6. 17:25

   In both of these movies, the protagonists, their weights are fixed, and their context windows gets wiped every single morning, and it's really problematic to go to work or have relationships when this happens, and this happens to LLMs all the time. I guess one more thing I would point to is security kind of related limitations of the use of LLMs. So for example, LLMs are quite gullible. They are susceptible to prompt injection risks.

7. 17:48

   They might leak your data, etc. And so, and there's many other considerations security related. So basically, long story short, you have to simultaneously think through this superhuman thing that has a bunch of cognitive deficits and issues. And yet, they are extremely useful. And so how do we program them, and how do we work around their deficits