A rejected NLP paper anticipated prompted language models

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 15:03

   Right. So you've been working in AI for a very long time, something like 20 years or longer. What do you think the people who have just gotten into AI in the last few years, basically post-ChatGPT, what is their worldview missing, do you think, if anything? um so it is interesting like ai went like when i got into it uh back in the day you know we had statistical pattern recognition machine learning deep learning artificial intelligence agi and si now and and the field has certainly changed from being like an interesting intellectual endeavor that is extremely important and lays foundation to you know try to understand the very nature of intelligence which is you know sort of what makes us most unique uh as as humans uh in in biology and so on, like all of that has shifted now.

2. 16:00

   You know, folks who just want to make some money and try to do that in crypto like three years ago and now try to do it with AI. So the whole field is shifting a lot. But on the positive side, of course, we are seeing now as folks have thought about and worked on AI and pushed the field forward on research for many years. It's incredibly fun to see all the applications really coming to fruition. And I've personally always been like a little bit lazy for some tasks that are just incredibly boring.

3. 16:26

   And I love the fact that I can offload some of those things now to an AI, though, of course, like as AI touches more and more human lives, like we have to think about like all the implications and how to do that well. And I think long term, I'm extremely optimistic and short term, there will be, you know, pushback like we've seen with pretty much every technology. We've seen like moral sort of outcries around like comic books and even novels like a couple hundred years ago and computer games and the Internet and cell phones and like all of these things.

4. 17:02

   i think eventually people will adjust uh to the change uh and then we'll be like of course we wouldn't go back to not having the internet or or comic books and so on um and in fact some of them may move into the canon of like what you teach uh kids in school um and yeah so education will have to update a lot too but i digress yeah when you were working on neural networks back in the 2000s did you think like oh this kind of thing like image net the technology behind image net is going to eventually become the precursor to like artificial general intelligence or was it like were you sort of narrowly thinking of it as like you know this is going to be really good for image recognition but um not necessarily for literal agi personally i've always been very excited about trying to unify everything in one model um and so it's very top-down um sort of thought that i had

5. 17:58

   pretty early on. And for me, the hard part was you had to publish papers in a certain way to be acceptable. And so every time I tried to publish papers that unified more things in one model, it was hard to publish. Even when we invented prompt engineering, it's a paper called DecaNLP, and you can actually see the reviews online still. People, the reviewers thought it was a ridiculously terrible idea to try to unify every single task of NLP and just prompt one model with a question of like, what's the sentiment of this tweet?

6. 18:37

   What's the summary of this document? What's the translation into German? But we showed it can be done with one neural net and just prompting it with different questions like that. The paper got rejected, but fortunately, Alec Radford, Ilya, and others picked it up, cited it like half a dozen times, and credited it with this idea that you can actually unify it. And that, you know, yeah, was cited in the early GPT papers and then had its impact through that.

7. 19:03

   But, yeah, it is a very counterintuitive solution in the early days of NLP and AI. Now, of course, it's like the opposite. it now it's so obvious you can't even invent it uh it's just like it's just breathing air you're just talking about you know asking questions to a model totally well it's been really interesting