In this study, the local thickness of the panel surrounding a muffler was optimized to improve the transmission loss of the muffler within the desired frequency range, and the results were verified using experiments. A rectangular muffler made of acrylic and aluminum, one side of which can be regarded as a vibrating panel and its other sides as rigid walls, was designed and used as the analysis model. An optimal design problem for the muffler was formulated considering that a change in the panel's vibration characteristics highly influences the acoustic characteristics of the expansion chamber constituting the muffler. The objective function was declared by reflecting a target noise frequency range and applying the vibro-acoustic coupling effect while using the local thicknesses of the panel as design variables. The muffler's optimal design problem was solved using an optimization algorithm, and a discussion conducted on the results with regard to vibro-acoustic coupling. Finally, the effectiveness of the designed muffler was verified. The results of this study show that the effective application of the vibro-acoustic coupling effect can significantly increase the noise reduction performance of a muffler.