I think its because of his smile and his eyebrows.

And he just "turns off" one of those when switching between types of roles.

He's got the maddest eyebrows in the biz, but the moment he cracks a smile, you can't help but to think he's a goofy s.o.b.