Yes, behavioral as well as physical differences are grist for the selective mill. I think socio-sexual selection has been seriously downplayed and overlooked. If someone is acting weird, you might not want to breed with them. If the behaviou4 set cones with physiochemical cues, it makes avoidance that much easier.

If the minority nevertheless manages to thrive and populate just enough to prefer hanging and mating with each other, and stop mixing with the parent group, then you have a new race, which might become a new species one day if the two groups drift genetically far enough away from one another to have sterile babies.