I hear you. I've been in exactly that position relatively recently.

Ultimately I made the choice to walk away from my only group of friends, basically the only people I spoke to regularly other than my mum and partner.

In a lot of ways, it was also the best choice I ever made.

It's a difficult change, but a necessary one.