It might be the genre.

I used to be the same and then I made a point to read more female authors and I've found several I like a lot. Toni Morrison and Yaa Gyasi are 2 that I particularly like. I've read 5 of Toni Morrison's books in the past year and loved each one. Also they are only like 200-300 pgs so not a huge commitment, maybe try one.