I would try Titebond ll glue with sawdust if I were to try. I like the glue as it has some extra strength and almost a vinyl like filler. I'd be more tempted to cut it out and make whole square pieces to fill it in (Tweeter mount).You could also have a larger rear plate for extra re-enforcement from the inside. I personally hate to work with particle board.
Having a carpentry background, I would just cut a new panel. Now the question is how hard is it to get that whole panel out?
Edit: Hey, Michael and I agree on the glue!