Maybe the single best depiction of the early stages of a Singularity, all disguised as a love story!

You're absolutely right. I certainly hope it works out that way.

I think Sparrow is really interesting. It's intentionally limited in order to fit a specific vision and be more effective in particular use cases.

But that also suggests that you could use the same techniques to create a huge range of different models for different purposes.

Good point. Of course, ultimately, superhuman kindness is exactly what we want in an AGI. However, I think the *appearance* of superhuman kindness in "companion" language models would just be another kind of superstimulus that a normal human couldn't compete with.

If you spend a significant amount of time interacting with an entity that never gets angry or irritated, dealing with regular humans could be something you would come to avoid.

I'd expect many people to have both. What I'm concerned about is how, eventually, human companionship might just not be very compelling compared to a good language model.

An "AI" partner has no needs of its own. It can be as endlessly loving or supportive or kinky or whatever as you need it to be. Once they can also give legitimately good advice I can imagine a lot of people finding real human relationships to be not much more than a pain in the ass. Human relationships are hard!

It has already started with apps like Replika.

At the moment, the human tendency to anthropomorphize is meeting language models halfway - but it won't be long until we're in Her territory. I'd expect many people to have a language model as their primary means of emotional support by 2030.

People are (correctly) alarmed by superhuman intelligence but I'm just as worried by superhuman charm, kindness, empathy and persuasiveness.

I doubt most governments have the imagination to grasp just how profoundly powerful AGI will be.

They're focused on guns, bombs, oil and money. It's the Maginot Line all over again.

PaLM's logical reasoning really blew my mind. That, more than anything, convinced me that we are close to AGI.

Sparrow seems like a proof of concept for Oracle-like weak AI.

You underestimate how quickly things have progressed.

It's possible that war could accelerate things.

What do you mean?

Thanks for your explanation. That makes more sense. Doesn't David Deutsch take a similar position?

That was exactly my point.

If you agree that a lower qualitative level of intelligence can't recognize a greater one, what makes you so confident that our level is "universal"?

Perhaps we can agree that a baby or small child, similar to animals, does not have universal intelligence. At what point do people "graduate" into it?

Are you saying that there is a specific line that separates "limited intelligence" from "universal intelligence" and that "mentally disabled" people (and presumably animals) fall on the limited side?

Where do you see that border? Do you have any evidence to back that up?

Personally, I'd love to believe that I have universal intelligence but I'm skeptical since I doubt that a lower level of intelligence is able to even recognize a level of intelligence sufficiently beyond it.

I think you are right. People won't even notice that systems like that will quietly deliver better and better answers until eventually they'll be solving problems that are currently outside of our grasp.