It's a slightly different skill to reading, and you just need practice. I suggest books where it won't hurt your enjoyment too much if your mind wanders a bit to start out with. Eventually you'll get just as good at "reading" with your ears as with your eyes, and you'll be able to listen to audio-books while doing chores or even shopping. I swap between listening and reading on a kindle, and if you held a gun to my head I wouldn't be able to tell you which parts I read and which parts I listened to.