- One AI Thing
- Posts
- 🦙 LLaMa, LLaMa, Open Source Drama
Hello, and thanks for reading One AI Thing. Get smarter about artificial intelligence, one thing at a time.
👀 Today’s Thing: LLaMa, LLaMa, Open Source Drama
No, not the children’s book.
🤖 Plenty of folks across industry, academia, and, of course, Reddit, are worried about a world in which big players like OpenAI, Google, and Microsoft own the most powerful AI models, have the (enormous) resources to maintain and update them, and basically control everything. Enter open source! A number of organizations have been announcing and building open source AI models in recent weeks and months. And they’ve got cute animal names like Dolly, Koala, and LLaMa, to boot! But not everyone thinks open source is the way to go. Unless they actually do.
🎧 Good AI models need good data. Listen to my conversation with David Kanter, Founder and Executive Director of MLCommons, about his organizations efforts to make more good datasets accessible to the public.
📖 Backstory
☞ Open source refers to software that was made to be shared, hacked apart, and improved upon by more or less whoever’s interested enough to do the work. When open source works, projects develop faster and yield better results thanks to the collective work — and intelligence — of everyone contributing to it.
☞ OpenAI originally launched in 2015 as a nonprofit with the aim of building value “for everyone rather than shareholders.” The company has since changed to a “capped profit” model, and the technical report accompanying GPT-4’s launch was purposely absent any meaningful details regarding the model’s development:
Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method.
☞ RedPajama, a collective effort led by Silicon Valley company Together, is an effort to fully open-source LLMs. RedPajama recently released a 1.2 trillion token dataset that follows the LLaMA recipe. RedPajama’s data enables any organization to pre-train models that can be permissively licensed.
🔑 Keys to Understanding
🥇 OpenAI leadership says their about-face on open vs closed is about competition and safety … but mostly safety. “If you believe, as we do, that at some point, AI — AGI — is going to be extremely, unbelievably potent, then it just does not make sense to open-source,” Ilya Sutskever, Co-Founder and Chief Scientist at OpenAI, said back in March.
🥈 Now, in the wake of a wave of open source LLMs inspiring researchers and startups to advocate for open source AI, OpenAI may be reconsidering. Chatbots and other AI apps built atop LLaMa and other open source models are rivaling ChatGPT and other commercial apps in quality. And, as noted at the top of this post, open source software by nature has the potential to develop faster thanks community involvement. Enough potential, in fact, to warrant a warning memo (that was subsequently leaked) from a Google engineer. So maybe it’s not a surprise that OpenAI is considering something of a re-pivot back to openness?
🥉 Ben Smith, VP of Information Design at Nomic AI: “For people to make informed decisions about where [GPT-4] won’t work, they need to have a better sense of what it does and what assumptions are baked in. I wouldn’t trust a self-driving car trained without experience in snowy climates; it’s likely there are some holes or other problems that may surface when this is used in real situations.”
🕵️ Need More?
Searching for a certain kind of AI thing? Reply to this email and let me know what you'd like to see more of.
Until the next thing,
- Noah
p.s. Want to sign up for the One AI Thing newsletter or share it with a friend? You can find me here.