What Google's Leaked Letter tells us about the AI Landscape [Finance Fridays]
How Meta's decision to share their work caused Google, Microsoft, Open AI and the rest of Big Tech to go on a multi-billion dollar wild-goose chase.
Hey, it’s your favorite cult leader here 🐱👤
On Fridays, you get posts about money 💲💰💰💲. Expect a mix of posts on personal finance, breakdowns of the most relevant events in the Tech Industry, and insights into the unique workings of the tech industry. 🔍🔍. Use these posts to master the money game, up your business sense, and solidify your finances to ride out the meanest of recessions.
To get access to all the articles, support my crippling chocolate milk addiction, and become a premium member of this cult, use the button below-
p.s. you can learn more about the paid plan here.
By now, most of you have probably heard about the recently leaked document by a Google Researcher commenting on the state of the AI Space and the Large Language Models Arms Race. The sentiments expressed in the article document paint a very dark picture of the so-called “Age of Foundation Models”- in stark contrast to the hype being propagated by VCs, AI Companies, Social Media Influencers, and even AI doomers. In this article, I will be going over the document and its implications in more detail. Stay tuned, because you definitely don’t want to miss out on this (we’ll follow up on this with a piece on how businesses can use open-source to thrive tomorrow).
We’ve done a lot of looking over our shoulders at OpenAI. Who will cross the next milestone? What will the next move be?
But the uncomfortable truth is, we aren’t positioned to win this arms race and neither is OpenAI. While we’ve been squabbling, a third faction has been quietly eating our lunch.
Key Highlights
Why Open Source is Winning- Firstly it’s free. Having been a creator for a while, I can tell you firsthand that a price tag, no matter how small immediately causes many people to turn away. The free and open nature of open source enables contributions from everyone, which allows for solutions to scale much quicker. However, that is not all. After all, this would not explain the rapid development in the open-source community and how they have been able to beat Big Tech companies with Million Dollar Budgets and multiple years of building Large Language Models-
Open-Source companies are doing things in weeks with $100 and 13B params that we struggle with at $10M and 540B.
There is one other thing that has allowed the open-source community to stack them Ws and pull off feats that Big Tech hasn’t been able to.
Open Source is more diverse- Think back to how Big Tech Hires people. And how it evaluates them. And how regimented practices are. All of these are great for efficiency, but it comes with a huge drawback- they kill innovation. The unfortunate side-effect of implementing systems to solve any problem is that your operations become more inflexible. There is a very strong selection bias in the kinds of people who will thrive in any regimented system. People who don’t conform to those ideals are filtered out- either in the hiring stages, through layoffs, or through employee churn (employees quitting because they don’t like the work/culture).
To a bird born in a cage, flying is a sickness
-We become blind to the flaws of the circumstances surrounding us.
Here’s an example- research positions in these companies generally require a Masters/Ph.D. degree. If most people in your research teams are from research team are from this background, they will replicate many of the flaws found in academia and upper-level education. And these will be invisible to the people in the system (and very hard to change because of the status quo bias). Once again, the Open Source community has a much lower barrier to entry- allowing more people to contribute. And when we put lots of different people together, and let them cook, great things happen. This is a clear example of that.
Are Language Models Dead- The document also made a claim that’s got a lot of AI Companies Sweating- “People will not pay for a restricted model when free, unrestricted alternatives are comparable in quality. We should consider where our value add really is.” This had a lot of people shaken but aligns well with what I’ve been saying for a while. Most recently, we had a discussion around AI Models and Moats on April 14th in a group chat I’m in. In it, I said the following about AI Moats-
I want to draw attention to the fact that this document specifically says "We Have No Moat, And Neither Does OpenAI". I’ve been calling this for months. This has now been backed up by the Google Researcher themselves. So what’s next for AI Models and companies looking to profit from these models? Just look at the last line- monetize the products around AI, not AI. There is a world of difference between AI Models and AI Solutions, and that difference is where money is made. Unfortunately, this is a lot trickier than it seems. If you’d like more input on that side, make sure you come in tomorrow for the piece on how you can use open source to help your business, or reach out to me to discuss your needs.
How Meta caused its rivals to go on a Billion Dollar wild goose chase- Around this time last year, Meta did something that no one saw coming- they decided to open source their research. The first thing they did was to release their OPT Model- their equivalent to GPT. Even though I covered it (and even called it the biggest development in AI), most people didn’t pay too much attention to the release. However, this has been the case. Take a look at two quotes from my analysis of the situation-
Firstly, since the model is open, it is possible for people to find and discover areas for improvement. This facet of the open-source culture is what is responsible for the explosive growth of tech over the last 2 decades. This gives them access to potentially millions of hours of free debugging/testing done by the community.
and
And the model being released for free also means that people are now much less likely to use paid models from their competitors.
These are both quotes from my article- Implications of Meta’s challenge to GPT-3 and Open AI, written on May 12th of last year. And both of these events played out exactly as I said. And while companies were investing billions into the LLM wars, Meta stayed out of the whole thing. Their competitors have burned billions, there is no profitability in sight (after spending 540 Million Dollars on building ChatGPT, Open AI is considering raising money through more funding), and now open source is lapping everyone up using resources that were made available from Meta. Here is what the document described as the problem with Google and OpenAI and why they were being beaten by OpenSource.
Wild Conspiracy theory- Meta leaked LLaMa themselves to break push open source community even further. Notice how the Google researcher’s assessment of the situation aligned almost exactly with what I said would happen. Between this, my calls on Crypto and LLMs being a bubble, my call on AI Emergence being BS, and my assessment that making ML Models bigger wasn’t useful- I have a much better track record on predicting tech than most of these so-called experts. Maybe I should quit Engineering and become an investor.
Overall, this document didn’t say much that should be shocking. And the fact that people are losing their minds over these claims shows how disconnected people in AI have become from common sense. If you’re someone/know someone who needs someone to tell shocking insights such as-
Retraining models from scratch is the hard path
Large models aren’t more capable in the long run if we can iterate faster on small models
Data quality scales better than data size
I have a secret 50,000 USD investing tip to sell you.
That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. If you like my writing, I would really appreciate an anonymous testimonial. You can drop it here. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow.
Upgrade your tech career with my newsletter ‘Tech Made Simple’! Stay ahead of the curve in AI, software engineering, and tech industry with expert insights, tips, and resources. 20% off for new subscribers by clicking this link. Subscribe now and simplify your tech journey!
Using this discount will drop the prices-
800 INR (10 USD) → 640 INR (8 USD) per Month
8000 INR (100 USD) → 6400INR (80 USD) per year (533 INR /month)
Reach out to me
Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.
Small Snippets about Tech, AI and Machine Learning over here
If you like my writing, I would really appreciate an anonymous testimonial. You can drop it here.
To help me understand you fill out this survey (anonymous)
Check out my other articles on Medium. : https://rb.gy/zn1aiu
My YouTube: https://rb.gy/88iwdd
Reach out to me on LinkedIn. Let’s connect: https://rb.gy/m5ok2y
My Instagram: https://rb.gy/gmvuy9
My Twitter: https://twitter.com/Machine01776819
" There is a very strong selection bias in the kinds of people who will thrive in any regimented system. People who don’t conform to those ideals are filtered out- either in the hiring stages, through layoffs, or through employee churn"... bravo!
The level of bias and conformity in science never ceases to amaze me, even from people who claim to fight against it.
"Retraining models from scratch is the hard path"... I'd say it's insanity. How could they build such models without a memory system for instance, when it's more than obvious that memory is a mechanism that allows to trade time against efficiency, and avoid relearning all the data on Earth with every new model? And still in a non efficient way...
I'd go even further: how can anyone believe it's possible to achieve human intelligence with statistics? Evaluating the probability of a word near other words, seriously? The word combinations complexity is infinite, so it will work only for the easiest things, or most obvious ones, and a ton of data (and they've exhausted all the data available already).
And that's exactly what we're seeing: chatGPT is very good at the most basic thing, ie chatting, conversing. Because the size of the training data is so huge, and language patterns are present everywhere, and above all, limited, so it converses well. It seems to perfectly 'understand' what we ask and formulate a perfectly worded answer.
I'm talking about the format, not the content. As soon as it becomes a bit complex, the accuracy drops, and sometimes becomes negative (hallucinations).
And, as you say, those teams of PhD's all suffer from the same issues, coming from academia. No one can see past his own beliefs, but in this case, it's a tragedy. They're out of touch with reality as you say. Inebriated by all that money and hype floating around.
As I often say, "it will never work because it cannot work".
Developing models that are basically mapping (ie limited right from the start) to learn reality, which has infinite complexity, should be cause for termination. Maybe it's better I'm not in charge :-)
I still don't understand why AI model itself is not a moat, because it is still too costly to unsupervised train a LLM base model. E.g. For now, Meta's LLaMA, it requires A100 (80GB) x 2048 to train a series of models (7b, 13b, 30b and 65b).
Most existing open source LLM models still relies on leaked LLaMA to supervised fine tune as new model. I.e. if LLaMA has no leak and the moat has not been removed, open source communities can only stick on their own self-trained low-performance 3b/7b, or even worst, achieve nothing.
My impression is: For now and near future, AI model itself is still a moat. For far-away future (>5 years), to be honest, I don't know. Or something I missed?