It is highly unlikely that the scope is at fault, particularly if it holds zero in between shots and during transport. This is likely a mechanical issue. You mentioned that the rifle returned to zero after removing the moderator/ re-adjusting the barrel.
I would keep remove the moderator and store the rifle as normal and after a day or two, test it out again. If there's no POI shift, repeat the test again with the moderator installed. I would also try both tests, but with the rifle's shroud/barrel not being under load during storage (for example, laying down in a case).
If there's a POI shift in either of the first two tests, but not when the rifle is stored horizontally, that indicates that some part of your shroud is taking a set from constant load. I'm not familiar with the XM1, but this could be due to a shroud being supported by soft o-rings that are compressed during storage.
The most likely candidate is just that screws on the shroud were loose. Shroud screws can often vibrate loose during shooting. I wouldn't loctite them, but you could try clear nail polish to give them a bit more resistance.