You can use a buffer material. I use 3/4 inch fiberglass ceiling tile between the actual crossover substate and the speaker side panels. You can keep the wire leads off the panels or use a staking compound that absorbs vibration like a pliable RTV.

Rob