- cross-posted to:
- technology@beehaw.org
- cross-posted to:
- technology@beehaw.org
A new paper suggests diminishing returns from larger and larger generative AI models. Dr Mike Pound discusses.
The Paper (No “Zero-Shot” Without Exponential Data): https://arxiv.org/abs/2404.04125
Not even another info transferring entity would solve it. Be it quantum computers, photonic computers, at the end of the day we’d be simply brute forcing the problem harder, due to increased processing power. But we need something else than brute force due to the diminishing returns.
Just to give you an idea. A human needs around 2400kcal/day to survive, or 100kcal/h = 116W. Only 20% of that is taken by the brain, so ~23W. (I bet that most of that is used for motor control, not reasoning.) We clearly suck as computing machines, and yet our output is considerably better than the junk yielded by LLMs and diffusion models, even if you use a really nice computer and let the model take its time producing its [babble | six fingers “art”]. Those models are clearly doing lots of unnecessary operations, while failing hard at what they’re expected to do.
Regarding research, my point is that what’s going to fix generative models is likely from outside the field of artificial intelligence. It’ll be likely something small and barely related, that happens to have some ML application.
there’s a lot to optimize in LLMs and i never said otherwise. Though, photonic computers if the field would be researched, could consume as much as an LED lamp making it even more effective than our brain. given the total amount of computers in the world, even the slightest power consumption optimization would save colossal amount of energy, and in case of photonics the raw numbers could possibly be unimagineable.
I bet they simply will find a way to greatly simplify the mathematical apparatus of the neuron interaction. Matrix multiplication is kinda slow and there’s lots of it