So true. That's why it's such a tough issue.

In the case where 2 powers are competing to reach AGI, there is a potentially huge first move advantage if one power reaches AGI before the other. So how do they each agree to slow development on safety grounds, when they also know there is a possibility of the other side disregarding their agreement to get a competitive lead? A kind of prisoner's dilemma, they both need to refuse the easy win.

Ultimately, I think who wins is less relevant than a safe win. The pendulum of power swings, and so long as humans are "in charge" in the grand scheme of things, it doesn't really matter (in my view) if they are outside my own sphere of values. So long as they have human values! The pendulum of cultural power will hopefully continue to swing and in a few centuries, the planet will still have humans on it.

Although there is a possible future where human directed AGI changes the world in a permanent negative way (against my own values) - like some sort of totalitarian dystopic power, but as a pessimist I don't think that outcome is exclusive to "the other side" winning this race. Both sides have the power to do bad things.