Nous grew from two Discord friends into an open-source lab

Transcript

Transcribed by machine (Whisper) from the episode’s audio, so a word may be off here and there.

1. 36:11

   So briefly, maybe we can talk about the origin story of Hermes. It's a bunch of nerds getting together for open source. What's the origin story? Yeah, absolutely. So I had been doing like a chat with PDF type of thing with people where I would go to a company and UGBT3 to do basic tool use to read their documents and have an AI chat pod they could chat with. At the time, many people were doing this. It was a lot harder to do then than obviously it's very easy to do now, but brand new stuff for us. And I was doing that solo while I was also volunteering somewhere called Open Assistant. Lion, who had made the pile, one of the biggest to kind of og image databases um they had been trying to do active rlhf with the community so they wanted to collect people's like rlhf uh on certain like traces they're they're like yes

2. 37:07

   on their preference data and i was helping quantize models they're not really doing anything crazy um and they had eight a100 nodes there and i had read the alpaca paper and i started synthing data and it changed the seed tasks and instead of using gbt 3.5 i used four and um technium was also doing the same thing and we were already friends on twitter and i messaged him and i said hey i have eight a100 nodes like uh do you want to train something together so we trained a gbt4x vicuna on the vicuna model using the data we made and it came out okay We got some people interested, like Emozilla, our CTO, Jeff.

3. 37:53

   But it was upon doing the same run on the base model, Llama base model, that we got huge interest from people, hundreds of thousands of downloads in just a few days. People starting to ask, what is news research? Now, news was just me and Technium at the time, just two guys hanging out in our little Discord server. People came to us, I won't name names on companies, but said, you know, you must be training on the benchmarks.

4. 38:18

   You must be training on the benchmarks. Technium had been coding for less than a year at the time, and I was a religion major in school. I didn't know. We asked these guys, what are benchmarks? We're doing this based off the heuristics that we understand are going to make models better from using them. We don't know about all this stuff. And so, of course, it got independently tested and found to be the best open fine tunes at the time, right?

5. 38:45

   2023, like mid-2023. So we did Hermes 1, Hermes 2. But after we make the first Hermes model, many, many people ask, who's newest research? You know, we want to get involved. And Technium and I decided, you know, this is our opportunity to bring together people to do open source volunteer work and really do open work now that GPT-3 is out and OpenAI has become closed. We want to continue to do this.

6. 39:11

   And Technium's goal for Hermes, by the way, that was the last Hermes I really worked on, the first one. After that, really, Technium has run it with the post-training team. He's done an amazing job. And he's also the initial creator of Hermes' agent. So he's the father of Hermes, really. And, you know, it was at that time that he had said, i want gbt at home i want chat gbt at home that's my north star a gbt4 at home and once that we got there you know it just kept going right like we kept going it's like we need to keep giving this level of intelligence to everyone we need to keep like letting everyone be on the even and equal playing field like the world needs to move and lockstep on this together and not just inequality gets created from this and so we formed a cohort 40 people or so of researchers, Jeff and Bowen come together, they make yarn, more and more work gets done. And we get reached out to, we get reached out, we get an email info at newsresearch.com. I just happened to make from Dylan. Dylan Rolnick is our CEO. Dylan Rolnick is, you know, basically a co-founder, he's an initial member of News with Us.

7. 40:25

   and he had seen what we were doing and he said i think that you guys have what it takes to be a full-time lab you know i see you guys are just volunteering but i think you could be a lab like let me contribute and help you become what you're meant to be and help you get the resources and help you get the access and go from a group of volunteers to a serious organization that's what this came in right and together we were able to take news from this group of volunteers grassroots uh to doing more and more with models to eventually making army's agent and being lucky enough to be here today you know none of us are trying to be steve jobs none of us think that we have some holy mandate of having to change the world or anything we just care like we just are guys who want this stuff available ourselves and we don't think that we

8. 41:18

   deserve it more than anyone else or less than anyone else so we we do this because like we would want someone to do it for us if we were on the other side so i guess the mission is to bring