In doing well. What about you, OP?

I think “pure” intelligence by itself is an algorithm. And as such an algorithm has no desire to do anything. Just like a web server has no desire to output web pages - it just does because it’s what the logic happens to make. It’s possible we need some sort of a quantum computer to achieve brain-like performance of intelligence - but I think intelligence is an algorithm specifically because LLMs exist.

Being able to pick an option is not necessarily a hallmark of cognition either - again a web server does that all the time. So I have a desire to think because it will end up getting me food and shelter and all the things I want. If we give a computer - a capability to think - it would have no desire to do so unless we executed the thinking algorithm.

Based on my research the intelligence by itself has no wants. Our wants come from being biological and evolving basic instincts and feelings like pain is bad, hunger, want to reproduce. All our other wants come from intelligence applied and predicting outcomes that maximize the biological wants. Without biological wants there’s not much to build off of.

I like this path of reasoning. Let’s assume whoever invents the AGI wants “greater good”. However malicious humans exist so how does that interact with AGI technology. Let’s say the governments started implementing some sort of universal safety net, but I am malicious and want to be have better status. What can I do?

I can start stockpiling still-scarce resources. With AGI that’s land and natural resources. I can made my own AGI that aligns with my own values and start a smearing campaign that degrades values I don’t like… but if everyone has AGI access the latter is probably not very effective. I can set up traps and bully others to show how much control I have to get people to fear me… I suppose AGI can amplify this but hopefully it amplifies the “good guys” as well so it cancels out at a societal scales but local cliques will still get hurt.

I guess land and natural resource hoarding become most viable forms of greed… I wonder if this wood make feudal societies a thing again. Did modern governments happen because intellectual labor was more profitable and so it was more profitable to share land and natural resources in order to have intellectuals work together?

Your statement implies AGI is good only for bad things (things I view as bad). That’s not the case as I see AGI being good for just doing things I don’t like doing. (AGI I’m asking about are not LLMs that we have now)

This is along the same lines I imagined it as well. And yes modern LLMs are nothing like what I’ve described. We’d need kilowatts of power per every single LLM conversation if we were to gradient descend them per conversation…. Funny enough human brain runs at like 10 watts. So I imagine there’s some huge paradigm shift from what LLMs do to what human brain does. You don’t get 1000 time efficiency boost from just putting all the world’s ram into it.

Hmm I see your points and I generally agree with them, except I don’t think we will ever collectively agree to ban AGI research before we invented it. There’s just too much juicy power for the governments to agree to ban it. As a result, it will happen eventually - maybe not in our lifetime but it will happen.

Oh I agree that “if these guys make agi” then we’re screwed. But I also don’t think Altmans and Amodeos of the world will invent AGI. However in that case the road isn’t smooth either. I dream of Star Trek like society but to get there is still going to be hard because it’s a disruption.

What about the benefits like having super cheap doctors available at your home at your schedule? You doing think the benefit is worth the change?

You just… invented dependency management.

The problem isn’t what can be done, the problem is what is being done. An AI in hands of a good developer can make that developer better - sure.

On the other hand what actually happens is equivalent to submitting code that doesn’t compile, except it’s not quite as easy to discover. The code does compile but it often is still bogus. Maybe it doesn’t solve the issue, maybe it does but in a round-about way, etc.

Legal and economical arguments aside I’m not saying “don’t use AI” - I’m saying “if you submit AI code - verify its work before making someone else verify its work”. And that means you need to know how to verify its work - just compiling and running isn’t enough.

I have no ethical problem with that but I can’t speak for the masses. But also it’s a bit in a grey area legally since it’s unclear who owns LLM outputs.

It doesn’t run doom, it plays doom. Very different. We don’t understand the neurons well enough to translate doom to neural networks, but we can train them to play doom, due some definition of playing.

The problem is that FOSS projects that are popular already have a a high quality standard because they are developed by very smart people who’s been developing for a long time.

There is inherently a problem with current AI - It doesn’t learn.

When a new developer joins a FOSS project they first spend weeks or even months learning this project’s specific coding practices, standards, and reasoning. I’ve been programming for 20 years and have a PhD. I can’t “just start contributing” to something like KDE because I’ve never looked at its codebase. I don’t know assumptions, methodology, or challenges that are baked into the codebase. I need to spend time to learn those to be a useful contributor.

AI can’t learn unless you do gradient descend. The problem that you mention about sounding elitist is that those FOSS developers spend a good amount of their time training new contributors and everyone is proud of their work. Now you have people who are not ready to be contributors tell AI to contribute and it’s doing its best but it just can’t learn and even having a 200K token context is not enough to fit all those project specific assumptions and rhetorics in.

No in the matrix, smart machines captured humans and used their body heat to work.

Here dumb humans captured useless human tissue in hopes to run dumb machines.

Pretty sure I managed to formally define intelligence and I’m working on implementing it on a computer. You essentially get an LLM that doesn’t need world’s books to talk. I suspect it’ll also not hallucinate because it’s lossless unlike neural networks. I’d say I’m about half way done implementing it.

Its an interesting perspective, except… that’s not how AI works (even if it’s advertised that way). Even the latest approach for ChatGPT is not perfect memory. It’s a glorified search functionality. When you type a prompt the system can choose to search your older chats for related information and pull it into context… what makes that information related is the big question here - it uses an embedding model to index and compare your chats. You can imagine it as a fuzzy paragraph search - not exact paragraphs, but paragraphs that roughly talk about the same topic…

it’s not a guarantee that if you mention not liking sushi in one chat - talking about restaurant of choice will pull in the sushi chat. And even if it does pull that in, the model may choose to ignore that. And even if it doesn’t ignore that - You can choose to ignore that. Of course the article talks about healing so I imagine instead of sushi we’re talking about some trauma…. Ok so you can choose not to reveal details of your trauma to AI(that’s an overall good idea right now anyway). Or you can choose to delete the chat - it won’t index deleted chats.

At the same time - there are just about as many benefits of the model remembering something you didn’t. You can imagine a scenario where you mentioned your friend being mean to you and later they are manipulating you again. Maybe having the model remind you of the last bad encounter is good here? Just remember - AI is a machine and you control both its inputs and what you’re to do with its outputs.