So in that specific case I'd say that's kind of silly to worry about. Unless you're a celebrity or have some specific factor that drums up weirdos, no one cares about your address. If somebody wants to be weird to someone, they don't need to find randos on the internet to do it - they can be weird to the multitude of people that live near them. It's not like human beings are a rarity that we need to dox people to track down.

Now personal info that can be used to steal someone's identity, that's another thing and people are always right to worry.