All in all, would it really be worth his time and money to sue? Does he have the means to begin with?

It’s not like you baited him in anyway to say these things. His comments were unsolicited.

The message was left on your equipment, so I would say the recording belongs to you.