I don't think you are fully considering the possibility of maliciousness. RdRand, because of on-chip whitening is completely opaque, there is no way to audit its functionality. How can we trust something like that with such crucial cryptographic functionality? If your seed your RNGs with predictable seed, then all of your crypto can be easily broken.
Sure, if CPU is backdoored, then your system is compromised no matter what you do, and it can leak all secrets in whichever way. Most of that can be detected post hoc or even heuristically. What is insidious about potential RdRand-based backdoor is the leak would take form of normal functionality, so there is no payload or suspicious communications to intercept and reverse-engineer.
I am not saying that RdRand should not be used, I am saying that RdRand should not be used in a way that makes system that easy to compromise. Why, for example, Ts'o did not use mixing function for this? Whole implementation reads like an entry into underhanded crypto competition.