I think some of the comments are misunderstanding OP's full question. They're asking, if the reproduction is difficult and the offspring are most likely sterile, how did species come to have different chromosomes at all? If we all have a common ancestor when you go back far enough, that must mean a mutation happened that caused one of them to have a different number of chromosomes. Most of the comments are saying how the actual producing offspring isnt difficult, but the fertility of that offspring is rare. But then how could those mutations in the number of chromosomes have become persistent enough that the offspring of them were fertile and able to even pass that down themselves?