That reason behind that perception is that distortion is perceived as being "louder", when in fact it's filled with all sorts of nasty harmonic artifacts that hopefully tells your ears to "turn it down"...because in reality....it's just plain unpleasant to listen to.
Ironically....it's also very unpleasant for the speakers to reproduce.
Again...to reiterate....
It's not the lack of power that burns up speakers. It's the distortion level.
Again...to reiterate. If you need...or like to listen at loud volume levels....you need the power to get there cleanly.
So...if you listen loud....get more power.
Practice "Safe Sound"
...or you'll eventually kill your speakers...tweeters and mids go first.