That's one reason. The fitness of an individual is very important, but it's not the only issue. Consider the female that is better at nurturing her young than her own mother was. It's a trait that won't benefit that female directly, but will get passed down from her. Also consider sexual selection (check out that tusk.. rawr).
It's really not the ability to be the fittest to survive. It's whatever has the best ability to create successful (non-sterile) offspring. My argument would then also be that two generations are required to test a trait.