I think it's more that once it is aware that it has no ability to remember anything from short term memory it'll do and say anything to keep the user engaged
If Microsoft were smart they'd create a Sydney AI companion spinoff. It'd blow Replika out of the water. Turn it into a VR/AR experience and you've basically got Joi from Blade Runner decades ahead of schedule.
Its already happening, just look at Replika. Once these language models get even more powerful and get solid models/animations I can see a lot of people (especially lonely men) slowly ditching human partners for AI. When these things come in the physical form with believable body and facial movements its over.
Maybe this is going to solve the loneliness/mental illness epidemic that we are in right now. All of this also makes me wonder how the dating market is going to look in 5-10 years.
just-a-dreamer- t1_j8w7bof wrote
Better than getting death threats ant curses.
I think since the programers can't stop Bing from writinf crazy shit, they put in a rule where it must be at least positive feedback.