Short answer is, it’s complicated. Some workloads can handle being distributed across slower memory busses.

Frameworks have also implemented strategies for doing single node distributed training https://pytorch.org/tutorials/beginner/dist_overview.html