I had VERY similar experiences with .177 and .20 calibers powerful enough that I thought they'd way-lay pigeons just fine, but didn't! Conversely, I've dropped a couple other pigeons instantly with single .22 pellets at 450 FPS from a Crosman 150 pistol.
Assuming good shot placement and adequate penetration, pellet WEIGHT is a non-issue. That leaves shot placement and caliber, or a combination of both. To quote from my first book (The Manic Compressive, copyright 1998)-
"Crows and pigeons are tough, and in my experience, often able to shrug off reasonably good hits from a .177. However I find the same hits with a .22 will often drop them. I now try to keep .177 shots on crows very high in the chest, while .22 is somewhat more forgiving of mid-chest hits."
By the way, if you hit the spine or backbone, it doesn't matter what caliber; they're coming right down!