I wouldn't absolutely bar myself from unrepping someone, but a) I'm not an Indian giver, and b) any failure of judgement in awarding rep is my own, and I wouldn't feel right having another pay the penalty for my error.
If I rep unwisely, that's my cross to bear.