AI labs plan to automate their own research first

Transcript

From the transcript The Diary Of A CEO publishes with the episode.

1. 19:32

   I keep mentioning superintelligence. Perhaps I should say more about like the sequence of events that the companies are planning to do. So right now they're focusing on automating coding. They're taking their AIs, they're making them bigger, they're training them for longer, and they're especially focusing the training on getting them to be good at autonomously writing and editing code. Because...

2. 19:55

   That will help the companies go faster, right? If they can automate the code, then they can do their own work better and faster and accelerate progress. The next step, which they've already begun, is to look at the rest of the research process as well. Coming up with ideas, analyzing experiments, communicating those results, all the other parts of the research process. They're trying to figure out how to train AIs to be good at those as well so that they can have AIs do the entire thing autonomously.

3. 20:24

   When you say do the entire thing, what do you mean do the entire thing? So like Anthropic and OpenAI in particular are trying to automate themselves. Like they're trying to make it the case that they don't really need human employees anymore. They just have a giant army of AIs that's churning away, doing all this autonomous research to make better AIs, to train the new AIs, put them in charge so they can make even better AIs and so forth.

4. 20:50

   And of course, not all just happening internally, but also interfacing with the world, going out and talking to people, collecting the data, setting up the training environments, doing the business deals, and so forth. They're trying to automate all of that. The reason why they're doing this is because they're trying to get to a position where they have... AIs that are superhuman at everything, superintelligence, and they're trying to get there before their competitors do. Needless to say, this is incredibly dangerous, I would say, you know? And in addition to being dangerous, it's a power grab, right? Like, if they actually succeed at this, then they'll be sitting on top of this army of superhuman AIs that will give them immense leverage over all sorts of other actors in the economy. Yeah.

5. 21:34

   insofar as they can work out something with the presidents and integrate it into the military or whatever, then that would give the U.S. immense hard power over all other countries, right? Obviously, nobody knows exactly when this is happening, but a very disquieting thing has happened over the last year to me, which is that when we published it in 2027, people were generally of the opinion that my timelines were too short and that like probably it would take more than 2027 until we got to the sort of events that I was just mentioning. You know, recursive self-improvement, AI is automating the whole research process, superintelligence.

6. 22:11

   These types of milestones, they happen in 2027. Which is this research paper you published. That's right. It's a scenario forecast that sort of lays out, like, month by month, a possible future trajectory. There was sort of like, at the time that we started writing, it was my best guess as to what would actually happen. Obviously, there was a lot of uncertainty, but, you know, I thought it's valuable to make a concrete guess just to sort of see what it might look like.

7. 22:37

   And at the time we were writing this, a lot of my friends in the AI industry and in nonprofits and so forth that work on AI, a lot of people were saying like, yeah, that stuff's going to happen, but like, it'll probably take a couple years longer than you think. And now it's more 50-50, especially when I go talk to people at Anthropica and OpenAI. They're often like... Yeah, no, 2027, that's basically what's going to happen. Just like you wrote. Why did you update your timelines? Oh, yeah, context for this is after writing it in 2027, I shifted my timelines to be a little bit more conservative. So at the time that we published, my 50% mark was in 2028, not in 2027. And then after we published, progress just seemed like it was going a bit slower. And so I updated to 2030.

8. 23:27

   which is still could happen sooner, could happen later, 2030. But now when I talk to people in the companies, they're like, it's not going to take that long. They're like, oh, you need to shorten them again. Like, get them back to 2027 or 2028, you know? So that's a bit disquieting. Again, don't know how long it's going to take, but this is the stated plans of the AI companies is to do this incredibly dangerous thing, and they think that they're just a few years away. So you wrote this report here, What 2026 Looks Like, and you wrote this in 2021, and it was remarkably accurate. Helped make a name for yourself amongst...

9. 24:04

   amongst everybody in AI. And which one was it that J.D. Vance, the vice president, read? I think it was this one, wasn't it? Yeah, this one. And then so then you published this one, AI 2027, and this was published, I believe, in 2025? Yeah. Yes, that's right. April. What were you forecasting in here? What are the key things that you said in here for people that haven't read it? The high level version of it is they automate decoding, then they automate the rest of the research process, then the pace of progress accelerates dramatically, they get to super intelligence, they're working with the government, specifically the president, the executive branch naturally wants to control this technology and otherwise wants to use it to beat China and integrate it into the military and so forth. By this point, it's sort of

10. 24:45

    doing basically all the work itself. I mean, it's super intelligent. So it's coming up with all these great ideas for how to integrate itself into everything and all these new technologies that's invented and so forth. And because of the race dynamics and because of the profit motive, they end up deploying it everywhere. And it builds robot factories, they build more robots, they build more robot factories, et cetera. Transforms the world entirely.

11. 25:05

    And then at some point it has enough power, it meaning the AIs, have enough power that they don't have to pretend to be aligned anymore. Right? Then they stop listening to orders. That's the race ending of AI-2027. We also wrote a sort of different branch, which is the slowdown ending, which was intended to sort of illustrate the concentration of power issues that I mentioned previously. So what if, hypothetically, the alignment issues get sorted out sufficiently quickly? Like, what if it turns out that, like, It's not too hard. With two months of slowdown, we can figure out how to make the AIs robustly do what we want and have the values that we want them to have. So that's one possible branch. And in that branch, it looks pretty similar. You know, they take the jobs, beat China, etc. But instead of the AIs ultimately killing everyone, they create this sort of amazing utopia. But the amazing utopia is...

12. 26:06

    whatever the people who controlled the AIs wanted it to be, right? And so that would be a very small group of people, like the president, some CEOs, etc. There should be a button just down below here. And if it says subscribe, you're already subscribed. If it says subscriber, that means you're not yet. And if you're not subscribed, please could you do us a favor and hit that button? It helps the show more than you know. And according to the algorithm, you're someone that watches our show, but you haven't yet hit that button. Thank you so much. Thank you.