Exactly, this is something many people seem to get confused by. survival of the fittest really means just that, the organism with the right mix of traits will win.
No, that's a common misconception.
The driving force is that the less fit will lose more often. This may sound like it's a different way of phrasing the same thing, but it isn't.
Say you have three variants of a species: one that can run 4 mph, one that can run 6 mph, and one that can run 8 mph. And a predator that can run 5 mph.
Evolution doesn't reward the fittest - it increases the risk of the least fit. Given a few generations, the first variant will dwindle, while the two other variants will intermingle and be just as successful.
Even worse, a predator may arrive that can run 10 mph or throw atl-atls. In which case the fittest doesn't survive, and the species might go extinct. One day something else may move in to fill the niche, but that's not a certainty.
What doesn't disadvantage an individual to a statistically significant degree isn't evolved away, even if it isn't the "fittest". Which is why men still have nipples, and we all have tailbones.
And being the fittest doesn't imply survival. It's not very useful against a direct meteor hit. Being less fit and somewhere else may be more beneficial.
So the saying really shouldn't be "Survival of the fittest", but "Higher death risk for the most unfit" doesn't roll as nicely off the tongue.