# AI can automate correctness checks in code review

Markdown copy of https://realalpha.app/s/ai-can-automate-correctness-checks-in-code-review/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Tibo Sottiaux](https://realalpha.app/builders/tibo-sottiaux/) · Tibo Sottiaux is an engineer who created Codex and head of Core Products & Platform at OpenAI. · Led OpenAI's launches of Codex, ChatGPT Work and the new Dots personal agent platform.
- Host: [Gergely Orosz](https://realalpha.app/people/gergely-orosz/) · Host, The Pragmatic Engineer · Spent four years at Uber heading mobile, web and backend teams; shipped part of its payments system.
- Show: The Pragmatic Engineer
- Episode: [Building Codex with Tibo Sottiaux](https://realalpha.app/v/sLSTM9znQNs/), Sep 9, 2026
- Moment: 47:35 to 52:09 of the episode (5 min), playing on https://realalpha.app/s/ai-can-automate-correctness-checks-in-code-review/; the episode from there: https://realalpha.app/v/sLSTM9znQNs/?t=2855

## Takeaways

1. Models can trace bugs through dependencies and incorrect documentation.
2. Security review is automatic and can block a pull request.
3. Human review can focus on intent and required invariants.

## About this segment

OpenAI's review models catch deep logic and security issues, and security flags can block pull requests. Human discussion can shift from checking code to agreeing on intent and system boundaries.

From “Building Codex with Tibo Sottiaux”, Sep 9, 2026. [Watch the full episode](https://realalpha.app/v/sLSTM9znQNs/)

## Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[47:41](https://realalpha.app/v/sLSTM9znQNs/?t=2861) where do you see humans still being or developers being involved in the review stage valuable? And where is it fine? Did you find it fine to hand it off to an agent? Yeah, the role of code review is changing. One of the early projects that I did on Codex was like working with research on developing a code review model that was going to be to a level where it can spot mistakes in logic and reasoning to a degree where it would require humans like, you know, multiple, potentially multiple hours to capture the same level of mistake because it requires like really digging like, you know, three, four levels deep into like the dependencies and like, you know, just understand that maybe the documentation actually was wrong and like the implementation of like the third party dependencies like different from what you expected.

[48:32](https://realalpha.app/v/sLSTM9znQNs/?t=2912) And so therefore your invariants are not upheld. And these things, it's just like, you know, unless you're an expert in that library, you wouldn't know. And therefore you have a bug. And so we developed like these code review models and, you know, we released them and now they're like the same level of like capability and like ability to spot these mistakes by doing like, you know, deep verification or like just part of the mainline models.

[48:54](https://realalpha.app/v/sLSTM9znQNs/?t=2934) Like when we benchmark them, it's like they're like superhuman in code review. And this is not just true for correctness. This is also true for security. for example, where you're capable of, like, reasoning across, like, you know, very, very complex things. And then, you know, coming up with, like, hey, you know, you have a critical security vulnerability here, which is now mandatory across, like, all of OpenAI pull requests. Like, we block pull requests from merging if, you know, we flag them with, like, a security issue. And this is, like, all automatic. And the role of code review now is, like, I think it was always about correctness. It was always about, you know, ensuring that things work. But it was also sort of like a little ritual for information exchange and, you know, bringing people on the same page

[49:32](https://realalpha.app/v/sLSTM9znQNs/?t=2972) and, like, you know, encouraging, like, a discussion, which ideally would have happened before, but sometimes it just only happens, like, around the code because once it's merged, it just actually runs in production and it's doing stuff. And then you have to maintain it. So there's, like, this social aspect to it as well. And so I think all of it is changing. Like, the correctness, the cybersecurity, the security, I think that will be automated.

[49:53](https://realalpha.app/v/sLSTM9znQNs/?t=2993) Really what we see and I see is there's this sort of really discussion around the intent that takes place around the pull request. It's like, what are you even trying to do? And is that the right thing to attempt to do? I think you can have that discussion outside of the pull request. It doesn't have to be around code. So maybe this helps crystallize where a discussion needs to happen versus where we did it because maybe we didn't have the type of tool that we have right now.

[50:20](https://realalpha.app/v/sLSTM9znQNs/?t=3020) Yeah, I think this is going to change. And it was like a forcing function because you have to have that discussion where like it's good to have that discussion before you merge it and it becomes production code but I think there are other ways to have these discussions and you know design things together and make sure that the intent is good and then the code doesn't matter as much and it's interesting because when I think back of all my code reviews like of course I have like memories where like it was great we had a good discussion or I learned something really interesting but a bunch of times honestly it was such a pain in the ass like I was trying to get my stuff you're pinged hey could you remove my code and like no right now i'm busy no i really need this to unblock me and then you context switch and then i feel it's always been like good and bad right so i feel

[51:05](https://realalpha.app/v/sLSTM9znQNs/?t=3065) whatever we do there will be always upsides and and downside but they're now they're just moving so i guess one upside is as an engineer you might have to not give your attention to just kind of basic stuff that doesn't need your input per se yes it saves time and progressively what we're going to see is also like you have an agreement on you know the box and the overall contract of what it's supposed to do and then you know what is inside the box as long as you have like strict guarantees in terms of resource utilization data access um security these kinds of things it's like what happens inside the box is you know it could be literally anything you just like don't really need to care and like really what you need to agree on is like what does the box actually do and what are the invariants that must be satisfied.

[51:52](https://realalpha.app/v/sLSTM9znQNs/?t=3112) And I think that is then worth, you know, having like a really good conversation on, you know, maybe assisted by your favorite agent. But then once you have that and you have that understanding, it's just like changing anything within the box is like, you know, doesn't require for this discussion.