It matters on my mood and what's going on in the manga sometimes during a fight scene I like certain songs and does something like this
Awesome fight scene + nice song =
Fucking Awesome
However, this can also work in reverse so if I'm reading and a song really out of place is going on I get pissed and my enjoyment drops and I have to turn the music off even when a time when a song would normally be good during this scene appears.
Now there are also manga's where you absolutely
need to listen to a certain song while something's going on for instance in 20th Century Boys you won't get the full enjoyment if your not listening to Bob Lennon (Rock/Guta Lala Suda Lala version) or 20th Century Boy by T-Rex while viewing some scenes like this
Spoiler Alert! Click to show or hide