Your intuition is pretty spot on - light and sound are both waves (light does have quantum behavior as photons sound doesn’t, but when they’re acting as waves they’re similar), and both can be absorbed by materials they pass through - depending on the material some wavelengths are absorbed more, and some wavelengths are absorbed less. Think like how you can hear a neighbor’s TV’s bass easier than speech, for instance.

What you’re experiencing in particular is that the wavelengths you can see and the wavelengths you can hear are impacted by typical construction materials differently. If you could see X-ray wavelengths you’d be able to see through walls just as easily as you can hear through them, for example. Conversely, if your walls were made with noise absorbing foam like are used in recording booths then they would be opaque to visible light and also “opaque” to audible sound waves.