Carrying a gun around creates an elevated risk of harm to everyone in the vicinity. A latent threat of violence is still harmful to a peaceful society, even if that threat is not ultimately acted upon.

Carrying something whose primary intended use is to kill other human beings without overwhelming need is something that a functional society quite justly punishes. Just say you're afraid of black people.