Copying Jev’s interface does not reproduce its reliability

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 53:07

   quote end of quote because that's you know a topic that's top of mind on x and everything so for anybody that hasn't followed so you guys have been out of stealth for three weeks And then in the three weeks, you've had basically everybody and their brother come up with something from Cloudflare to Databricks to, I think, Microsoft just last night, Satya announced a Decision One category. What do you make of that?

2. 53:34

   And what do you say to the naysayers? The naysayers of us? Yeah, the people that think that you were just completely copied by everyone in three weeks, and therefore there's nothing left, which obviously is not the narrative, but I'm curious, like, how you position it? i think we will really see um like i don't want to punch down too much but i will like describe my take of things uh when i first started this project i literally thought it would take a week like i and i ended up with something very similar to what people are doing but the problem is it was really really clearly not robust and that was something that has like plagued me for years um and i get the sense that this is a thing and to be clear i'm a machine learning researcher so all my self-deprecating comments all those deprecating comments apply to me i was an arrogant

3. 54:29

   person who thought algorithms are the way to go and like oh yeah i can just you know do these shape rotations myself and make this happen um so i was able to make things like this in the first week of this project as well and the problem was at least in my experience at the time is it was so unbelievably dumb that it was embarrassing like getting the shape of something and and like copying an existing interface is very very easy um similar to what google did with flan back in the days of like rlhf like we actually had an early alpha they actually published before us they made like a version of it and it was not just bad it was worse than nothing um so things like this can happen all the time and i i'm curious about measuring how good they actually are from like an intellectual perspective as far as i can tell from the field they probably are

4. 55:24

   the equivalent of years behind us in terms of like actual reliability but i want to measure this before i like say that to anyone else and you're not a you're not a fan of benchmarks either in terms of like benchmarks are useless what was your post again the benchmarks lies and benchmarks or something like that yeah so dance marks are like like they really really hurt users and they actually hurt most the users who they think they're meant to help which is the users who do not have the time to play with things themselves like because they are the ones who are most misled by So I get the sense that everyone else is really optimizing for benchmarks, and that is, I believe, probably a bad strategy.

5. 56:10

   But again, we have better ways of measuring intelligence, so we plan to do this. My guess is that they are dumb and non-robust, but maybe that will be sufficient for some people's works. And if they solve people's problems, I am super jazzed for that. like actually i'm inspired by solving people's problems my only fear from this and benchmark maxing is that it might convince people that this entire category of models doesn't work because they use one of the dumb models instead of our model and my request for people is like you don't have to believe in us just try our model too uh at least and stick with what works um because what actual there's like a lot of nuance to making intelligence reliable that machine learning researcher types who really want credit and all of that truly do not understand

6. 57:01

   um and that's why we don't hire those types yeah it's you know playing playing back this conversation in in in my mind it looked like there's a lot of um smart people uh in the community for sure uh you know i mentioned certain companies obviously open ai uh released their own product called decisions just uh just a few days ago but playing back this conversation uh in my mind it seems that uh what you guys are doing is is very fundamental you know this whole discussion we had around rlhf and why that created a problem and how you removed it and replaced it with something and your rlcd and you need specific data for that that seems like a lot of work so what did people do then in these two weeks when you say interface they just took the results of a regular nllm and removed the text and forced it into some kind of probability exactly but they

7. 57:50

   didn't do the work to get there to just remove or to get it to be reliable and if this solves people's problems i am super duper jazzed for that and actually i do want to shout out opening eyes decision api for better or for worse they did not include benchmarks so they are a lot better than everyone else in my box so in ideal world they don't try to no one tries to mislead users but some companies need the attention and hopefully they don't create too much ill will.

8. 58:20

   And at the very least, people try our models as well so that they can figure out what actually works for them. But if the intelligence and reliability that we have is not necessary, I think use the other models. That's great. I want problems solved and I want automation to occur.