One way to achieve that is syncretism. The practice of putting parts of the existing religion into the new one.

The people kept celebrating their old festivals the way they used to but they had to put some new labels on some things.

Some deities and stories were identified with figures and deities from the new religion. So nothing much changed.

This is a feature of religions in general and happens naturally especially if there is no central authority that says what is and isn't true. It can also happen on purpose.

Another factor is that in most places most of the time religion was not just a thing that stood by itself. It was part of politics and daily life.

The government and the religious leadership were not really separate things, but so deeply entwined that they were almost the same.

Keeping the old faith was the same as keeping to the old rulers. If the ruler themselves converted publicly this became even less a thing people would do.

Freedom of religion was usually not a thing. People were whatever religion they ruler said they were and going against that was like rebelling against the ruler. It was rebelling against the laws and customs that held society together itself.

Rulers were not keen on that sort of thing and neither were most of the people themselves. Finding someone in their midst who went against their rules and customs like that would not even always require the intervention of some far away tyrant to resolve but would be done away with locally by the community.