I have to be honest, I now have two Crowns and have never had this problem. I don't think I treat them particularly gently, but I also don't throw them around by the barrel. There are a few things of note though which may cause perceived POI shifts that aren't actually the fault of the barrel/mount.
The factory shroud causes POI shift, retracted or extended. If you rotate it, you can watch your POI walk around on the target. I can only assume this is why it has an alignment pin for the extended position, and FX's assumption is that you'll transport it with the shroud retracted and extend the shroud and lock it in with the pin before shooting it the rest of the time. I personally prefer a moderator, so I simply moved the shroud stop rearward until it locked the shroud fully back in a single rotational orientation.
Rifle cant is another one. We're all familiar with how rifle cant works, however our flexible (by firearm standards) airgun barrels exacerbate this when we put extra weight on them. Meanwhile our popular brands of moderator are actually quite heavy and flex the barrel SIGNIFICANTLY. I test a lot of airgun silencers, and so watch them cause downward POI shift all the time, not due to velocity changes, but due to them flexing the barrel downward. That is all well and good, but now lean your rifle over a bit. Instead of your usual cant-related-drift, you've added a compounding factor of the moderator now flexing the barrel in a slightly different direction as well. So keeping your gun dead nuts vertical is important.
Wind drift is another one. Nobody wants to admit that wind drift is messing with them, but it is a thing to keep in the back of your head when looking for problems and solutions. Breezes which don't even rustle the leaves can still move pellets a surprising amount at 100 yards.
I hope that is helpful, not patronizing.
Also none of the smooth twist barrels I have had needed cleaning to stay accurate while the STX barrels often need regular cleaning so the POI doesn't change or the groups open up.
I had this issue when I first got my .22 Crown. It is hard to say what cured it exactly as I tried a lot of different things. The "final" three things I did though were: two good scrubbings with JB Bore Paste, cleaning with Tuff Glide, and then putting maybe a thousand rounds through the barrel without caring a lick about accuracy (silencer testing). By the time I popped out the other side of that I had expected the gun's accuracy to be a disaster and it'd be in desperate need of cleaning. To my astonishment the gun was shooting better than it ever had. I still clean it occasionally, but have largely stopped as it frankly doesn't seem necessary anymore. I'm not sure if what happened to me can/should be replicated, but I think it is at least an interesting anecdote that I started on your side of the fence where I had to clean a lot to maintain accuracy, and ended up on the other side of the fence where now cleaning is extremely infrequent. *shrug* I can't explain it.
