Rude to wear headphones around the house?

I find it really annoying when I'm tryingto talk to my friend only to realize he is listening to music. But, I think it's ok if you're wearing large headphones that are in plain sight and telling people that you tend to do that. In your case, turn the volume down when in a place where people would walk in and assume you would be listening.