Humans process language multi-modally. We don't just predicts the next word (although we do this as well), we also visualize. We decode language as images projected at an internal screen that we're not consciously aware of (read Louder Than Words by B. Bergen on that). We can imagine 2 as two objects, 3 as three, imagine all kinds of transformations and rotations of said objects and use all kinds of internal shortcuts to do arithmetic.

Or we can take a calculator and use that. It's another thing that language models lack, even though they're run on a "computer".

I believe when AIs will be given these capabilities, they will do math "out of the box" no problem.