I'm not sure if the idea of a contributor license as you suggest is in the spirit of open source.
Depends on which "spirit of open source" you are referring.
The classical Richard Stallman model is about giving the person running the code the ability to modify it for their own use and share those modifications with others. I believe that one of his first big gripes had to do with printer drivers, as an example. In that case, the contributor license agreement causes no harm at all. The CLA defines restrictions for the user if the user wants his contribution to become part of the codebase distributed by the original author. That does not stop the user from re-distributing the same codebase with or without his own changes (ie. fork).
In most cases, the CLA will never actually be necessary. Most new projects get few, if any, contributors. The summary even says the works they previously posted just sit there and rust - they aren't maintaining them, and neither is anyone else. On any cases where someone contributes code, it will either be:
a) very trivial, and thus no copyright transfers are needed (ex. a bugfix changing 2 if statements)
b) a small patch.
c) a large patch.
For the large patches, you'll just need to decide if the CLA is worth it or not. Is it a feature that is worth having. If so, is it worth the legal hassle, or should you just rewrite it?
For the small patches, those often need rewritten anyway. People rarely submit complete patches with tests and everything... so just rewrite it. If it's something that doesn't need rewritten, see the previous sentence. Contributors will often provide the patch with a public domain license if asked, which you can then relicense as needed.