" There is a very strong selection bias in the kinds of people who will thrive in any regimented system. People who don’t conform to those ideals are filtered out- either in the hiring stages, through layoffs, or through employee churn"... bravo!
The level of bias and conformity in science never ceases to amaze me, even from people who claim to fight against it.
"Retraining models from scratch is the hard path"... I'd say it's insanity. How could they build such models without a memory system for instance, when it's more than obvious that memory is a mechanism that allows to trade time against efficiency, and avoid relearning all the data on Earth with every new model? And still in a non efficient way...
I'd go even further: how can anyone believe it's possible to achieve human intelligence with statistics? Evaluating the probability of a word near other words, seriously? The word combinations complexity is infinite, so it will work only for the easiest things, or most obvious ones, and a ton of data (and they've exhausted all the data available already).
And that's exactly what we're seeing: chatGPT is very good at the most basic thing, ie chatting, conversing. Because the size of the training data is so huge, and language patterns are present everywhere, and above all, limited, so it converses well. It seems to perfectly 'understand' what we ask and formulate a perfectly worded answer.
I'm talking about the format, not the content. As soon as it becomes a bit complex, the accuracy drops, and sometimes becomes negative (hallucinations).
And, as you say, those teams of PhD's all suffer from the same issues, coming from academia. No one can see past his own beliefs, but in this case, it's a tragedy. They're out of touch with reality as you say. Inebriated by all that money and hype floating around.
As I often say, "it will never work because it cannot work".
Developing models that are basically mapping (ie limited right from the start) to learn reality, which has infinite complexity, should be cause for termination. Maybe it's better I'm not in charge :-)
I still don't understand why AI model itself is not a moat, because it is still too costly to unsupervised train a LLM base model. E.g. For now, Meta's LLaMA, it requires A100 (80GB) x 2048 to train a series of models (7b, 13b, 30b and 65b).
Most existing open source LLM models still relies on leaked LLaMA to supervised fine tune as new model. I.e. if LLaMA has no leak and the moat has not been removed, open source communities can only stick on their own self-trained low-performance 3b/7b, or even worst, achieve nothing.
My impression is: For now and near future, AI model itself is still a moat. For far-away future (>5 years), to be honest, I don't know. Or something I missed?
" There is a very strong selection bias in the kinds of people who will thrive in any regimented system. People who don’t conform to those ideals are filtered out- either in the hiring stages, through layoffs, or through employee churn"... bravo!
The level of bias and conformity in science never ceases to amaze me, even from people who claim to fight against it.
"Retraining models from scratch is the hard path"... I'd say it's insanity. How could they build such models without a memory system for instance, when it's more than obvious that memory is a mechanism that allows to trade time against efficiency, and avoid relearning all the data on Earth with every new model? And still in a non efficient way...
I'd go even further: how can anyone believe it's possible to achieve human intelligence with statistics? Evaluating the probability of a word near other words, seriously? The word combinations complexity is infinite, so it will work only for the easiest things, or most obvious ones, and a ton of data (and they've exhausted all the data available already).
And that's exactly what we're seeing: chatGPT is very good at the most basic thing, ie chatting, conversing. Because the size of the training data is so huge, and language patterns are present everywhere, and above all, limited, so it converses well. It seems to perfectly 'understand' what we ask and formulate a perfectly worded answer.
I'm talking about the format, not the content. As soon as it becomes a bit complex, the accuracy drops, and sometimes becomes negative (hallucinations).
And, as you say, those teams of PhD's all suffer from the same issues, coming from academia. No one can see past his own beliefs, but in this case, it's a tragedy. They're out of touch with reality as you say. Inebriated by all that money and hype floating around.
As I often say, "it will never work because it cannot work".
Developing models that are basically mapping (ie limited right from the start) to learn reality, which has infinite complexity, should be cause for termination. Maybe it's better I'm not in charge :-)
I still don't understand why AI model itself is not a moat, because it is still too costly to unsupervised train a LLM base model. E.g. For now, Meta's LLaMA, it requires A100 (80GB) x 2048 to train a series of models (7b, 13b, 30b and 65b).
Most existing open source LLM models still relies on leaked LLaMA to supervised fine tune as new model. I.e. if LLaMA has no leak and the moat has not been removed, open source communities can only stick on their own self-trained low-performance 3b/7b, or even worst, achieve nothing.
My impression is: For now and near future, AI model itself is still a moat. For far-away future (>5 years), to be honest, I don't know. Or something I missed?