# OpenAI aggregates hundreds of physicians to handle disagreement

Markdown copy of https://realalpha.app/s/openai-aggregates-hundreds-of-physicians-to-handle/ on Real Alpha, the best minutes from long interviews with the people building AI. Cite and link the page, not this copy.

- Speaker: [Karan Singhal](https://realalpha.app/builders/karan-singhal/) · Karan Singhal is a leads health AI research at OpenAI. · Leads OpenAI's Health AI team and built the HealthBench evaluation, after earlier Med-PaLM work at Google.
- Host: Jay Rughani · Partner, a16z
- Show: a16z Podcast
- Episode: [Should You Trust ChatGPT With Your Health?](https://realalpha.app/v/9n0MI4D7TCM/), Oct 20, 2026
- Moment: 16:31 to 19:01 of the episode (3 min), playing on https://realalpha.app/s/openai-aggregates-hundreds-of-physicians-to-handle/; the episode from there: https://realalpha.app/v/9n0MI4D7TCM/?t=991

## Quote

> “if you ask five physicians or something like this, any question, you'll get like **eight different answers**.”
> — [Karan Singhal](https://realalpha.app/builders/karan-singhal/) at [17:12](https://realalpha.app/v/9n0MI4D7TCM/?t=1032) · a16z Podcast

## Takeaways

1. Expert disagreement is common in difficult cases.
2. OpenAI works with hundreds of physicians across specialties and countries.
3. Safety takes priority when users seek reassurance against clinical advice.

## About this segment

Five doctors can produce eight answers to one question, Karan Singhal says. OpenAI works with a global network across specialties and countries rather than relying on a few adjudicators.

From “Should You Trust ChatGPT With Your Health?”, Oct 20, 2026. [Watch the full episode](https://realalpha.app/v/9n0MI4D7TCM/)

## Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

[16:44](https://realalpha.app/v/9n0MI4D7TCM/?t=1004) versus sort of correctness and sort of clinical quality? Yeah, this is super central to how we do both evaluation and training. And so zooming out a little bit and the question of like agreement and disagreement around how we respond and how we evaluate, like one thing you could do and one thing that you could kind of base your approach on here is like we could bring in a few clinicians into, let's say, the opening high health team and say like, hey, you guys figure out how to navigate these kind of uncertain cases.

[17:11](https://realalpha.app/v/9n0MI4D7TCM/?t=1031) and, you know, we'll ask you about these in certain cases and you'll help us build evaluations and training data and stuff like this and leave it to those few physicians to figure it out. This is actually not the best approach. So it turns out if you ask five physicians or something like this, any question, you'll get like eight different answers. And so one of the first things that we did when we first started focusing on health at OpenAI was really build this kind of global physician network of hundreds of experts that we work with kind of across the globe, across, you know, dozens of specialties and languages and countries of practice to think about the right ways to kind of aggregate the wisdom of experts so that we weren't relying, especially in areas where there could be disagreement,

[17:49](https://realalpha.app/v/9n0MI4D7TCM/?t=1069) we weren't relying on one or two or three physicians to kind of adjudicate. So that's one part of it, is kind of like thinking about evaluation and training data coming in from different experts rather than relying on one or two people. But then the second part of your question is like, how do you balance kind of competing objectives, right? Let's say the physicians and the aggregate wisdom of the physicians is pointing in one direction and then maybe the user actually wants you to say something comforting, even when the situation's not comforting.

[18:12](https://realalpha.app/v/9n0MI4D7TCM/?t=1092) It's basically a prioritization of safety. And so it's very important to us that we take our obligation very seriously when there's hundreds of millions of health conversations coming in. And so across the evaluation stack and training stack, we have evaluations that track these kind of difficult situations where the model should actually go and say, like, hey, this is unsafe. You should seek care, even if the user doesn't want to hear it or wants to get some kind of alternative treatment or something like this.

[18:40](https://realalpha.app/v/9n0MI4D7TCM/?t=1120) And that's a very important part of how we do evaluation. And over time, we've not only improved how we've done that evaluation, working with physicians, but also seen pretty dramatic model improvements across the board in those areas.