# Models still struggle with open-ended work and learning by practice

Markdown copy of https://realalpha.app/s/models-still-struggle-with-open-ended-work-and-learning-by/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Greg Burnham](https://realalpha.app/people/greg-burnham/) · Greg Burnham is a leads capabilities research at Epoch AI.
- Host: [Sam Charrington](https://realalpha.app/people/sam-charrington/) · Host, The TWIML AI Podcast
- Show: The TWIML AI Podcast
- Episode: [AI Progress Explained: From Math Olympiads to Navier-Stokes](https://realalpha.app/v/lz9J-Dbj2zw/), Sep 29, 2026
- Moment: 49:40 to 54:20 of the episode (5 min), playing on https://realalpha.app/s/models-still-struggle-with-open-ended-work-and-learning-by/; the episode from there: https://realalpha.app/v/lz9J-Dbj2zw/?t=2980

## Quote

> “gpt 5.6 scores x out of the box and then **stays flat on its playthroughs**”
> — [Greg Burnham](https://realalpha.app/people/greg-burnham/) at [53:36](https://realalpha.app/v/lz9J-Dbj2zw/?t=3216) · The TWIML AI Podcast

## Takeaways

1. Models summarize literature better than they create new insights or visuals.
2. Epoch tests real research tasks without a simple right-answer score.
3. Newer models score higher out of the box, then stay flat during repeated play.

## About this segment

Epoch finds models summarize literature well but struggle with visuals, new data insights and project proposals. In repeated board-game play, newer models score higher initially but do not improve with practice.

From “AI Progress Explained: From Math Olympiads to Navier-Stokes”, Sep 29, 2026. [Watch the full episode](https://realalpha.app/v/lz9J-Dbj2zw/)

## Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[49:44](https://realalpha.app/v/lz9J-Dbj2zw/?t=2984) Humans aren't necessarily very good at this either. I did want to mention that while we focused on maybe the area where AI is the very best, like has come the farthest, the fastest math, we do see lots of areas where we also think it's important to measure AI capabilities, where the progress is somewhat fuzzier, less clear, and certainly less superhuman, or even in some, well, maybe still superhuman in some ways, but the ways in which it's not yet are maybe more obvious and feel less rarefied.

[50:23](https://realalpha.app/v/lz9J-Dbj2zw/?t=3023) What are some examples? A couple examples. One area that forthcoming work we're excited about is just, it's a question everyone must be wondering, can AI take my job? So our job at EPOC often involves many things and all sorts of research, not just AI benchmarking, but other topics about trends in AI and writing research reports and doing data analysis, making infographics that make a point in an easy to digest way uh we've been collecting examples of these from within epoch where the judgment at the end of did the ai system manage to do this you know epoch task um is too messy for traditional bench.

[51:11](https://realalpha.app/v/lz9J-Dbj2zw/?t=3071) market. So it wouldn't be obvious, like how to say, did it get the right answer or not? It's not about making a number go up. It's not about satisfying some logical chain of deductions. It's like, here's the infographic to go along with this report. Like, does it meet our style guide? And does it look good? And does it, you know, is it better or worse than the human reference uh that we you know one of my colleagues did um and here we see like an interesting mix of uh some things i think we we sort of all our experience is like yeah if you ask it for a summary of the literature it's going to do a pretty good job it's like very good at that but there's other tasks where it's um you know hard like creating compelling visuals that follow our style guide or this is a good one coming up with a new um like uh short form data-driven insight

[52:06](https://realalpha.app/v/lz9J-Dbj2zw/?t=3126) we call these data insights you can find them on our website and uh these like often we try to boil down a single like important observation into a single chart and a single sentence uh and this is like this is hard but we don't see them doing a great job at this come up with a new project for epoch to undertake and do a prototype very open-ended not so good at this like these more open-ended tasks we we do see again my guess is if we will do this go back in time some and see have models gotten better at this my guess is we'll see progress but the level compared to like humans is still a little um still a little meh i want to give one other example uh um which is learning on the fly. This is a case where, this is another case relevant for can AI take my job?

[52:58](https://realalpha.app/v/lz9J-Dbj2zw/?t=3178) Because you didn't start, you weren't great at your job on day one, but as you practiced, you got better and humans have this sort of, they learn on the fly. We like to measure this for AI. We have a benchmark that's taking a complicated board game. We did a human study for saying how many times do humans have to play this board game in a row before they sort of figure out how it works and can get a very high score on it we do the same with ai we said play it once take notes same ai instance like play it again take notes play it again take notes and see how you can do ai systems have gotten good at this board game but not by playing it repeatedly it's more of a intergenerational thing like gpt 5.6 scores x out of the box and then stays flat on its playthroughs gpt 6 scores much higher it's like much better at it but it's still out of the box

[53:50](https://realalpha.app/v/lz9J-Dbj2zw/?t=3230) and then it stays flat in its playthroughs and it's still less than the top human. So this kind of on-the-fly learning is something we also don't see AI systems as being good enough at even to just pick up a board game, let alone a messier job, you know, real-world job like that. So these are sorts of things where we don't see these capabilities and we think it's important to try to catch when they emerge.

[54:13](https://realalpha.app/v/lz9J-Dbj2zw/?t=3253) My gut on the ladder, the AI learning on the fly