It is not deterministic, it picks randomly, with the distribution skewed towards more probable words. That is what makes them so good - the difference between deterministic and stochastic AI doing this is the difference between GPT and predictive text on your phone. The predictive text always picks the most probable word leading to run on and repetitive sentences GPt has the ability to pick a less likely word which allows it to write much more varied and niche things.

It being stochastic is also the reason you can have it regenerate responses.