It was a burr that turned into a chunk off what I believe is the hammer. Pulled it out after realizing the retainer was just a plastic clip in. Got some fine grit sandpaper, rolled it up and got the inner part of the metal tube it slides in, then smoothed the chunk on the hammer down. Some light WD40 and I'm back in business!
Upvote 0