It does matter within reason, views on intimacy can be dealbreaker. Also stuff like infidelity in the past can most certainly be a dealbreaker, saying it doesn't matter removes all accountability from the decisions we made. If your partner isn't comfortable with your past then you weren't right for eachother in the first place.