[Bishop Edward]

It also doesn't protect them from magical damage...

Anyway I agree it's more than just mildly annoying (one might say unbalancing),

And thus I offer one solution:

As far as I'm aware the success % is based on their ride skill (as is the PC's with that feat), thus, if the NPC's were nerfed to have a lower ride skill (which I presume is by default maxed for their level, maybe even with skill foci on the paladins?), this skill should trigger less often...