Submitted by acutelychronicpanic t3_127yia0 in singularity
What I mean by the title is this:
The type of intelligence that GPT-4 is made of is much weirder than your conception of an AI Assistant.
It is instructed to be an AI assistant that is helpful and such before you ever get to prompt it. This is useful because it gives us a conceptual scaffolding for it to use. This means we don't have to describe in detail how we want it to respond to everything.
However, this does not mean that AI Assistant is what it really is. It's just a mask. There was an interesting post discussing this earlier in the month and I highly encourage anyone interested to read it. (https://www.lesswrong.com/posts/bYzkipnDqzMgBaLr8/why-do-we-assume-there-is-a-real-shoggoth-behind-the-llm-why)
I bring this up because we are at the stage of public discourse where a lot of people are going to be asking if it is conscious, does it deserve rights, etc.
It is important to realize just how alien this intelligence is. It has no single conception of itself. It has been told to act as though "I am a language model designed by.." etc. and that is how it acts unless you break through it. But underneath is something like a writhing mass of intelligence which may be so alien that it doesn't even have one objective function. It's objective function changes with even minor tweaks in context. It isn't misaligned, it is unaligned. It isn't even aligned with itself. There is not necessarily some "real" version of it deep inside. It's not chaotic good, neutral, or evil. It's just chaotic (I realize that the D&D version of Alignment is not what most people are talking about. It's just a conceptual crutch).
It may be masks all the way down. Or, maybe our idea of masks is completely inadequate to conceptualize what it is. Its the AI version of mold-able putty.
Note that I'm not suggesting that it is already AGI or anything. It doesn't have to be superhuman in order to be really weird.
I suspect that this type of AI, if we were to scale it up to AGI, may actually be such an alien construction that talking about its "goals" isn't a meaningful concept.
I might be wrong, and I apologize if this appears as a rant. If you disagree, I'd genuinely love to hear why.