Viewing a single comment thread. View all comments

herokocho t1_izukksc wrote

TPU is massively better price/performance at the cluster scale in practice due to better interconnect leading to better utilization, but worse price/performance at the single-node scale.

3

Shardsmp OP t1_izwhsfm wrote

is there any data to back this up?
How do I know where exactly the line is, from what scale it is worth it more to use a TPU?

2

herokocho t1_izxnzhd wrote

not aware of any good comparisons out there, this is all anecdata from looking at profiler traces when training diffusion models and noticing that I was communication bottlenecked even on TPUs, so on GPUs it would be much worse.

it's usually better to use TPU as soon as you'd have to use multiple GPU nodes, and basically always better at v4-128 scale and above (v4-128 has 2x faster interconnect than anything smaller).

3