I understand the by-the-book purpose of the apostrophe; it's just my personal thing that I do, knowing it's wrong but it looks more "right" to me, in a strictly visual sense, because you don't ever see a number touching a letter anywhere else.

If I wasn't using numerals I'd obviously write "eighties" and not "eighty's."