Best you can do is try to explain how it just slipped out (I've been there) and hope he understands that you just want to continue what you have (maybe feelings grow later).

If you feel that you'll never love him that way, don't keep him on the hook. I know it sucks, but you'll have to be blunt with him.