But that's kinda my point. Anything ejected would only "push off" the asteroid and help nudge it off course, given that they mostly get ejected either in the opposite direction to the spacecraft, or perpendicularly in a roughly even manner (this last one is speculation on my part since I imagine it depends heavily on the angle of impact and possibly more things I don't know, but I don't think it's a wild assumption)