The C4.1 contribution protocol I eventually wrote for ZeroMQ solved this problem. You have to develop rules that catch bad actors (yet not learners) and then educate project managers on how to fire people when needed.
Our rules for instance ask that you solve one problem with one patch, that you never break existing stable APIs, that you respect style guidelines, and so on. When people break these rules we give them several chances to improve their behavior. If they persist in doing it wrong, we remove them.
Turns out, when the rules are very explicit and teach people how to make good patches, then it's very rare we have to fire people.
The rules are at http://rfc.zeromq.org/spec:22