Cleaning Quiet Energy shroud

I was wondering if dirty parts affected sound reduction or accuracy. 

I recently removed the QE components after about 2000 shots and they where completely caked in dust. I washed the two baffles and only blew off the heavy stuff from the hair curlers and fiber cloth. 

Should I wash everything and replace the cloth or is that good enough for general maintenance?