Oversized bullets in Dragon Claw?

I shot my new Dragon Claw .50 for the first time yesterday. I had some .498 round balls, and some old T/C Maxi Balls. The 370 grain maxi balls is close to what I'm looking for to hunt deer and hogs. They shot really well, but I found out later that they are .504 and the rifle's bore is supposed to be .4985. I was unable to push them under the loading gate like the balls, which were easy. After shooting the first one, I kept going, and ended up shooting around thirty of them. One inch at 50 yards and I was happy. Should I have any concerns about shooting such an oversized bullet?
 
I have done the same. I even thought about getting a bullet sizing die for my reloading press to size them down a bit, but it doesn't seem worth the cost. Those muzzleloader conicals sell for like $1 per shot. .498 round balls from Track of the Wolf are the cheapest I've found, but I did score a .498 round ball mold from ebay recently. Neilson's appears to be sold out of everything, but his EPP's shoot good out of mine.