>give it some time before going back.

Absolutely not! Don't let someone else's interpretation of your remarks dictate your reality. Just keep going back as often as you normally would. Let them adapt to you, rather than you adapting to them. If you'd been an asshole, that would be different, But you were not being an asshole. You avoiding visits there makes it appear that you are remorseful about something. Please don't support that mistaken perception.